Home › Glossary › Evaluation › MMLU

Beginner · Evaluation

MMLU

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. Massive Multitask Language Understanding — a 57-subject multiple-choice benchmark spanning humanities, STEM, law, and medicine.

Technical Definition

Massive Multitask Language Understanding — a 57-subject multiple-choice benchmark spanning humanities, STEM, law, and medicine.

How it works

MMLU (Hendrycks et al., 2020) contains ~16k questions in subjects like high-school math, US foreign policy, and professional law. Models are scored 5-shot. It became the headline benchmark for LLM general knowledge and was near-saturated by GPT-4 (~86%), prompting follow-ups like MMLU-Pro and GPQA. Despite known label noise it remains a standard reporting metric.

Related Concepts

  • Benchmark — A standardized dataset and evaluation protocol used to compare models.
  • Evaluation — The process of measuring a model's quality or comparing different models.
  • HumanEval — OpenAI's 164-problem Python coding benchmark where models must write a function passing hidden unit tests — pass@k is the standard metric.