Advanced · Research

Emergent Abilities

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. New capabilities that appear in large models only after they reach a certain scale, not predictable from smaller versions of the same model.

Technical Definition

New capabilities that appear in large models only after they reach a certain scale, not predictable from smaller versions of the same model.

How it works

Emergent abilities are surprising and non-linear performance gains observed in very large AI models. These capabilities, such as advanced reasoning, in-context learning, or complex problem-solving, are not present in smaller models and cannot be simply scaled up. They suggest that increasing model size and data might unlock qualitatively new behaviors and intelligence.

Related Concepts

• Large Language Model (LLM) — A massive neural network trained on vast text corpora to understand and generate human language with remarkable fluency.
• In-context learning — A technique where a language model learns to perform a task by conditioning its output on examples provided within the input prompt.
• Scaling Laws — Empirical power-law relationships predicting how LLM loss decreases as model size, dataset size, and compute increase together.