Advanced · Research

Bayesian Optimization

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. An adaptive hyperparameter search that uses a probabilistic model to choose the next configuration to try.

Technical Definition

An adaptive hyperparameter search that uses a probabilistic model to choose the next configuration to try.

How it works

BO fits a surrogate model (often a Gaussian process) over past results, then picks the next configuration that balances exploring uncertain regions and exploiting promising ones. Particularly effective when each training run is expensive, like tuning large models.

Related Concepts

• Hyperparameter Tuning — The process of finding optimal configuration values that control model training, such as learning rate, batch size, and architecture choices.
• Grid Search — Exhaustively trying every combination of a predefined set of hyperparameter values.
• Random Search — Sampling hyperparameter configurations at random instead of testing every combination.