Beginner · Evaluation
Fraction of successes
Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.
TL;DR. A metric that measures the proportion of generated outputs that meet a specific success criterion.
Technical Definition
A metric that measures the proportion of generated outputs that meet a specific success criterion.
How it works
The fraction of successes is a simple evaluation metric used when the desired output of a model is binary (success or failure). It's calculated by dividing the count of successful outputs by the total number of outputs generated. This metric is particularly useful for tasks with clear, verifiable outcomes, such as code generation or solving mathematical problems.
Related Concepts
- Natural language generation (NLG) — The process of transforming structured data into human-readable text.
- Generative AI — An AI field focused on creating models that can generate novel and complex content, such as text, images, audio, and video, that is coherent and original.