Beginner · Evaluation

HumanEval

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. OpenAI's 164-problem Python coding benchmark where models must write a function passing hidden unit tests — pass@k is the standard metric.

Technical Definition

OpenAI's 164-problem Python coding benchmark where models must write a function passing hidden unit tests — pass@k is the standard metric.

How it works

HumanEval (Chen et al., 2021) ships each problem as a docstring and signature; the model completes the body and is judged purely by whether the generated code passes the held-out tests. Pass@k estimates the probability that at least one of k samples passes. It is the long-standing yardstick for code LLMs, supplemented today by MBPP, LiveCodeBench, SWE-bench, and Aider's polyglot benchmark.

Related Concepts

• Benchmark — A standardized dataset and evaluation protocol used to compare models.
• Evaluation — The process of measuring a model's quality or comparing different models.
• MMLU — Massive Multitask Language Understanding — a 57-subject multiple-choice benchmark spanning humanities, STEM, law, and medicine.