Intermediate · Research

Emergent Abilities

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. Capabilities like multi-step arithmetic or instruction following that appear suddenly above a scale threshold rather than improving smoothly.

Technical Definition

Capabilities like multi-step arithmetic or instruction following that appear suddenly above a scale threshold rather than improving smoothly.

How it works

Wei et al. (2022) showed that on certain benchmarks LLMs perform at chance until a critical parameter or compute scale, then jump to high accuracy. Examples include modular arithmetic, IPA transliteration, and chain-of-thought benefits. Later work (Schaeffer et al., 2023) argues some apparent emergence is an artifact of discontinuous metrics; the debate remains active and shapes how labs forecast frontier capability.

Related Concepts

• Large Language Model (LLM) — A massive neural network trained on vast text corpora to understand and generate human language with remarkable fluency.
• Chain-of-Thought (CoT) Prompting — Asking models to show step-by-step reasoning before giving a final answer, improving accuracy on complex tasks.
• Scaling Laws — Empirical power-law relationships predicting how LLM loss decreases as model size, dataset size, and compute increase together.
• Grokking — A training phenomenon where a network memorizes the training set quickly but only generalizes much later, after many additional optimization steps.