Intermediate · Fundamentals

Ensemble Methods

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. Techniques that combine multiple models to produce better predictions than any single model alone.

Technical Definition

Techniques that combine multiple models to produce better predictions than any single model alone.

How it works

Ensemble methods aggregate predictions from multiple models to reduce errors. Bagging trains models on random data subsets and averages outputs — Random Forest is the classic example. Boosting trains models sequentially, each correcting the previous model's errors — XGBoost dominates tabular ML. Stacking trains a meta-model on top of base model predictions. Ensembles reduce variance (bagging), bias (boosting), or both (stacking).

Related Concepts

• Overfitting — When a model learns noise and specific patterns in training data too well, causing it to perform poorly on new, unseen data.
• Bias-Variance Tradeoff — The tension between a model's ability to fit training data (low bias) and its ability to generalize to new data (low variance).
• Cross-Validation — A technique that evaluates model performance by training and testing on different subsets of the data in rotation.