Intermediate · Fundamentals

Hyperparameter Tuning

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. The process of finding optimal configuration values that control model training, such as learning rate, batch size, and architecture choices.

Technical Definition

The process of finding optimal configuration values that control model training, such as learning rate, batch size, and architecture choices.

How it works

Hyperparameters are settings defined before training begins — unlike model parameters (weights) which are learned. Examples include learning rate, number of layers, dropout rate, and batch size. Tuning methods range from grid search (trying all combinations), random search (sampling randomly), to Bayesian optimization (using past results to guide future trials).

Related Concepts

• Gradient Descent — An optimization algorithm that iteratively adjusts model parameters by moving in the direction of steepest decrease of the loss function.
• Learning Rate — A hyperparameter that controls how large each parameter update step is during gradient descent optimization.
• Cross-Validation — A technique that evaluates model performance by training and testing on different subsets of the data in rotation.