Intermediate · Evaluation
Inter-rater agreement
Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.
TL;DR. A measure of the consistency or agreement between two or more human raters performing the same task.
Technical Definition
A measure of the consistency or agreement between two or more human raters performing the same task.
How it works
Inter-rater agreement, also known as inter-annotator agreement, quantifies the level of consensus among multiple human annotators or raters. High agreement suggests that the task is well-defined and the raters are applying the criteria consistently. Low agreement may indicate ambiguous instructions or subjective task elements, potentially requiring refinement.