Beginner · Fundamentals
Gradient Descent
Visual diagram · Math · Worked example · 3 difficulty levels.
TL;DR. Gradient descent is the optimization algorithm that makes AI learn. It adjusts model parameters step by step to minimize errors.
Intuitive Explanation
Imagine you're blindfolded on a hilly landscape and need to find the lowest valley. You feel the slope of the ground beneath your feet and take a step downhill. Repeat this process — always stepping in the steepest downhill direction — and you'll eventually reach a valley. That's gradient descent.
Technical Definition
Gradient descent is a first-order iterative optimization algorithm for finding a local minimum of a differentiable function. It updates parameters θ in the direction of the negative gradient: θ_{t+1} = θ_t - η∇L(θ_t), where η is the learning rate and ∇L is the gradient of the loss function.
How it works
Gradient descent is the engine that drives nearly all neural network training. **Variants:** - **Batch GD:** Computes gradient over the entire dataset. Stable but slow and memory-intensive. - **Stochastic GD (SGD):** Uses a single random sample per update. Noisy but fast, with implicit regularization. - **Mini-batch SGD:** Compromise — uses batches of 32-512 samples. The practical standard. **Adaptive Methods:** - **Momentum:** Adds a velocity term to accelerate through flat regions: v_t = βv_{t-1} + ∇L; θ = θ - ηv_t - **RMSProp:** Adapts learning rate per-parameter using a running average of squared gradients - **Adam:** Combines momentum and RMSProp. Default choice for most practitioners: m_t = β₁m_{t-1} + (1-β₁)∇L, v_t = β₂v_{t-1} + (1-β₂)(∇L)², θ = θ - η·m̂_t/√(v̂_t + ε) **Learning Rate:** - Too high: overshooting, divergence - Too low: painfully slow convergence, getting stuck - Solution: learning rate schedulers (cosine annealing, warmup, step decay) **Challenges in Neural Networks:** - Loss landscape is non-convex with many local minima and saddle points - Saddle points are more problematic than local minima in high dimensions - SGD noise actually helps escape sharp minima, finding flatter (better-generalizing) regions
Mathematical Notation
θₜ₊₁ = θₜ − η · ∇L(θₜ)θ represents the model parameters, η is the learning rate (step size), and ∇L(θ) is the gradient of the loss function. Each iteration moves θ in the direction that most reduces the loss.
Visual Explanation (flowchart)
Initialize θ → Compute Loss L(θ) → Compute Gradient ∇L → Update θ = θ - η∇L → Converged? → [No: Loop] / [Yes: Stop]
Real-World Use Cases
- Google Brain: Research on Adam, AdaFactor, and other optimizers used across all Google AI products
- OpenAI: Training GPT models uses AdamW (Adam with weight decay) optimizer
- Hugging Face: All model training in the Transformers library defaults to gradient descent variants
Related Concepts
- Backpropagation — An algorithm that efficiently computes gradients by propagating errors backward through the network using the chain rule.
- Loss Function — A mathematical function that measures how far the model's predictions are from the actual values, guiding the learning process.
- Learning Rate — A hyperparameter that controls how large each parameter update step is during gradient descent optimization.