Intermediate · Evaluation
Automatic evaluation
Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.
TL;DR. Automatic evaluation uses software to assess model output quality, often by comparing it to a known correct answer.
Technical Definition
Automatic evaluation uses software to assess model output quality, often by comparing it to a known correct answer.
How it works
Automatic evaluation involves using algorithms or software to quantitatively assess the quality of a machine learning model's output. For tasks with predefined correct answers, this can involve direct comparison; for more complex outputs or generative tasks, specific metrics or even other AI models (autoraters) may be employed.
Related Concepts
- BLEU (Bilingual Evaluation Understudy) — A metric to evaluate machine translation quality by comparing generated text to human references based on N-gram overlap.
- Human evaluation — Assessing model performance by using human judgment to rate its outputs.
- Evaluation Metrics — Quantitative measures used to assess the performance, accuracy, and quality of AI models for specific tasks.