Advanced · Research
Scaling Laws
Visual diagram · (in preparation) · Math · Worked example · 3 difficulty levels.
TL;DR. Empirical power-law relationships predicting how LLM loss decreases as model size, dataset size, and compute increase together.
Technical Definition
Empirical power-law relationships predicting how LLM loss decreases as model size, dataset size, and compute increase together.
How it works
Kaplan et al. (2020) and Hoffmann et al. (2022, Chinchilla) showed that test loss follows L ≈ L∞ + (Nc/N)^α + (Dc/D)^β over many orders of magnitude. Chinchilla's compute-optimal recipe — roughly 20 tokens per parameter — corrected earlier under-trained giant models. Scaling laws guide every major training run today, determining model size, data budget, and learning-rate schedule.
Mathematical Notation
L(N, D) ≈ L_∞ + A/N^α + B/D^βRelated Concepts
- Large Language Model (LLM) — A massive neural network trained on vast text corpora to understand and generate human language with remarkable fluency.
- Compute — The processing power and resources required to train or run machine learning models.
- Emergent Abilities — Capabilities like multi-step arithmetic or instruction following that appear suddenly above a scale threshold rather than improving smoothly.