Intermediate · Fundamentals
Regularization
Visual diagram · (in preparation) · Math · Worked example · 3 difficulty levels.
TL;DR. Regularization is a set of techniques that prevent AI models from overfitting by discouraging overly complex solutions.
Intuitive Explanation
Think of regularization as Occam's Razor for AI. Given two explanations that fit the data equally well, prefer the simpler one. Regularization adds a 'complexity penalty' that nudges the model toward simpler, more generalizable solutions — like a teacher telling a student: 'Explain this simply, don't overcomplicate it.'
Technical Definition
Regularization refers to techniques that constrain or penalize model complexity to reduce generalization error. Common forms include adding a norm penalty to the loss function (L1/L2), stochastic methods (dropout), and implicit regularization through training procedures (SGD noise, early stopping, data augmentation).
How it works
Regularization is essential for training models that generalize well to unseen data. **Explicit Regularization:** - **L2 (Ridge / Weight Decay):** L_total = L_data + λΣw². Encourages small weights, creating smoother decision boundaries. In neural networks, implemented as weight decay in the optimizer. - **L1 (Lasso):** L_total = L_data + λΣ|w|. Encourages sparsity — many weights become exactly zero, performing automatic feature selection. - **Elastic Net:** Combines L1 and L2: λ₁Σ|w| + λ₂Σw² **Stochastic Regularization:** - **Dropout:** Randomly sets neuron outputs to zero during training with probability p. Forces the network to learn redundant representations. At inference, weights are scaled by (1-p). - **DropConnect:** Drops individual weights instead of neurons. - **Stochastic Depth:** Randomly drops entire layers in deep networks. **Implicit Regularization:** - **SGD noise:** Mini-batch gradient noise acts as implicit regularization, preferring flat minima. - **Early stopping:** Limits the effective model complexity by halting training before convergence. - **Data augmentation:** Increases effective dataset size, reducing overfitting. - **Batch normalization:** Adds noise through batch statistics, providing regularization effect. **Choosing Regularization Strength (λ):** Use cross-validation to find the optimal λ. Too small = overfitting persists. Too large = underfitting.
Mathematical Notation
L_reg = L(θ) + λ‖θ‖₂² (L2) or L(θ) + λ‖θ‖₁ (L1)The regularization term λ‖θ‖ penalizes large weights. λ controls the strength of regularization — too high and the model underfits, too low and it overfits. L2 penalizes the squared magnitude (smooth), while L1 penalizes the absolute value (sparse).
Real-World Use Cases
- OpenAI: Weight decay (L2) is crucial for training large language models like GPT
- Google: Dropout is used extensively in computer vision and NLP models
- Meta AI: Stochastic depth regularization used in training very deep residual networks
Related Concepts
- Gradient Descent — An optimization algorithm that iteratively adjusts model parameters by moving in the direction of steepest decrease of the loss function.
- Loss Function — A mathematical function that measures how far the model's predictions are from the actual values, guiding the learning process.
- Overfitting — When a model learns noise and specific patterns in training data too well, causing it to perform poorly on new, unseen data.
- Batch Normalization — A technique that normalizes layer inputs across a mini-batch, stabilizing and accelerating deep network training.