Intermediate · Research

Benchmark

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. A standardized dataset and evaluation protocol used to compare models.

Technical Definition

A standardized dataset and evaluation protocol used to compare models.

How it works

ImageNet, GLUE, MMLU, SWE-bench, and HumanEval shaped entire research subfields by giving teams a common scoreboard. Benchmarks accelerate progress but can also distort it when models overfit to the test set or when the benchmark doesn't reflect real use.

Related Concepts

• Accuracy — The fraction of predictions a model gets correct on a labeled dataset.
• Baseline Model — A simple reference model used to put more sophisticated approaches in context.
• Reproducibility — The ability for others to re-run an experiment and obtain the same results.