Intermediate · Fundamentals

Optimizer

Visual diagram · (in preparation) · Math · Worked example · 3 difficulty levels.

TL;DR. An algorithm that updates model weights during training to minimize the loss function, with strategies beyond basic gradient descent.

Technical Definition

An algorithm that updates model weights during training to minimize the loss function, with strategies beyond basic gradient descent.

How it works

Optimizers determine how model weights are updated. SGD is the simplest. Momentum adds velocity to smooth updates. Adam combines momentum and per-parameter learning rate adaptation, making it the most popular default. AdamW adds proper weight decay. More recent optimizers like LAMB and Adafactor target large-scale training.

Mathematical Notation

Adam: mₜ = β₁mₜ₋₁ + (1-β₁)gₜ,  vₜ = β₂vₜ₋₁ + (1-β₂)gₜ²

Adam maintains two moving averages: mₜ (mean of gradients) and vₜ (variance). These are bias-corrected and used to scale the update.

Related Concepts

• Gradient Descent — An optimization algorithm that iteratively adjusts model parameters by moving in the direction of steepest decrease of the loss function.
• Backpropagation — An algorithm that efficiently computes gradients by propagating errors backward through the network using the chain rule.
• Learning Rate — A hyperparameter that controls how large each parameter update step is during gradient descent optimization.