Intermediate · Fundamentals

Bias-Variance Tradeoff

Visual diagram · (in preparation) · Math · Worked example · 3 difficulty levels.

TL;DR. The tension between a model's ability to fit training data (low bias) and its ability to generalize to new data (low variance).

Technical Definition

The tension between a model's ability to fit training data (low bias) and its ability to generalize to new data (low variance).

How it works

Bias measures how far off predictions are on average from the true values — high bias means underfitting. Variance measures how much predictions fluctuate across different training sets — high variance means overfitting. The tradeoff arises because reducing one often increases the other. Simple models have high bias, low variance. Complex models have low bias, high variance. The sweet spot minimizes total error (bias² + variance + irreducible noise).

Mathematical Notation

E[(y - ŷ)²] = Bias²(ŷ) + Var(ŷ) + σ²

Total expected error decomposes into squared bias, variance of predictions, and irreducible noise σ². You cannot eliminate σ², but you can balance bias and variance.

Related Concepts

• Loss Function — A mathematical function that measures how far the model's predictions are from the actual values, guiding the learning process.
• Overfitting — When a model learns noise and specific patterns in training data too well, causing it to perform poorly on new, unseen data.
• Regularization — Techniques that constrain a model's complexity to prevent overfitting and improve generalization to unseen data.