Intermediate · Evaluation
Autorater evaluation
Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.
TL;DR. A hybrid evaluation method combining human judgment with an ML model trained to mimic human evaluators for generative AI quality.
Technical Definition
A hybrid evaluation method combining human judgment with an ML model trained to mimic human evaluators for generative AI quality.
How it works
Autorater evaluation is a technique used to assess the quality of generative AI outputs by blending human and automated assessment. An 'autorater' is itself a machine learning model trained on human-provided evaluations, aiming to replicate human judgment. While pre-built autoraters exist, fine-tuning them to specific tasks yields the best results for efficient and consistent quality assessment.
Related Concepts
- Generative AI — An AI field focused on creating models that can generate novel and complex content, such as text, images, audio, and video, that is coherent and original.
- Human evaluation — Assessing model performance by using human judgment to rate its outputs.