Beginner · Fundamentals

Learning Rate

Visual diagram · (in preparation) · Math · Worked example · 3 difficulty levels.

TL;DR. The learning rate controls how big of a step the model takes when updating its parameters during training — too big causes instability, too small causes slow learning.

Intuitive Explanation

Imagine tuning a radio dial to find a station. If you turn the dial in huge jumps, you'll skip right past the station. If you turn it in tiny increments, you'll find it but it takes forever. The learning rate is the size of each turn — finding the right balance is crucial.

Technical Definition

The learning rate η ∈ ℝ⁺ is a scalar hyperparameter that scales the gradient update: θ_{t+1} = θ_t - η∇L(θ_t). It controls the magnitude of parameter changes per optimization step, directly affecting convergence speed, stability, and final solution quality.

How it works

The learning rate is often called the single most important hyperparameter in deep learning. **Effects:** - **Too high:** Loss oscillates or diverges. Parameters overshoot optimal values. - **Too low:** Training is extremely slow. May get stuck in poor local minima. - **Just right:** Smooth convergence to a good solution. **Learning Rate Schedules:** - **Step Decay:** Reduce LR by a factor every N epochs (e.g., ×0.1 every 30 epochs) - **Cosine Annealing:** LR follows a cosine curve from max to min. Very popular. - **Warmup + Cosine:** Start low, ramp up, then cosine decay. Used in Transformer training. - **One-Cycle:** Ramp up then ramp down. Often finds good solutions faster. - **Reduce on Plateau:** Automatically reduce LR when validation loss stops improving. **Learning Rate Finding:** The LR Range Test (Smith, 2017): Train for one epoch, exponentially increasing LR from very small to very large. Plot loss vs LR. The best LR is slightly below where loss starts increasing rapidly. **Per-Parameter Learning Rates:** Adam and similar optimizers adapt the effective LR per-parameter using gradient statistics. But the base LR still matters enormously. **Practical Tips:** - Start with 3e-4 for Adam, 0.1 for SGD with momentum - Always use a schedule (cosine is a safe default) - Use warmup for Transformers (1-5% of total steps) - Larger batch sizes can tolerate (and may need) larger LRs

Mathematical Notation

θ ← θ − η · ∇L(θ)

η scales the gradient ∇L(θ). A typical starting value is 0.001 for Adam optimizer or 0.01 for SGD.

Real-World Use Cases

• Google Brain: Warmup + cosine scheduling is standard for training large Transformers
• fast.ai: Popularized the one-cycle policy and learning rate finder
• OpenAI: GPT training uses careful warmup and cosine decay schedules

Related Concepts

• Gradient Descent — An optimization algorithm that iteratively adjusts model parameters by moving in the direction of steepest decrease of the loss function.
• Backpropagation — An algorithm that efficiently computes gradients by propagating errors backward through the network using the chain rule.
• Loss Function — A mathematical function that measures how far the model's predictions are from the actual values, guiding the learning process.