Home › Glossary › Neural Networks › Batch Normalization

Intermediate · Neural Networks

Batch Normalization

Visual diagram · Math · Worked example · 3 difficulty levels.

TL;DR. Batch normalization stabilizes neural network training by normalizing layer inputs within each mini-batch, enabling faster training and higher learning rates.

Intuitive Explanation

Imagine a factory assembly line where each station receives materials of wildly different sizes and temperatures. Production would be chaotic. Batch normalization is like a quality control step between stations that standardizes the materials — same temperature, same size — so each station can work efficiently.

Technical Definition

Batch normalization normalizes each feature in a mini-batch to zero mean and unit variance, then applies a learned affine transformation: BN(x) = γ · (x - μ_B)/√(σ²_B + ε) + β, where μ_B and σ²_B are the batch statistics, and γ, β are learnable parameters.

How it works

Batch normalization (Ioffe & Szegedy, 2015) was one of the most impactful practical innovations in deep learning. **How It Works:** 1. Compute mean and variance across the batch for each feature 2. Normalize: x̂ = (x - μ) / √(σ² + ε) 3. Scale and shift with learnable parameters: y = γx̂ + β 4. At inference: use running averages instead of batch statistics **Why It Helps:** - **Smoother Loss Landscape:** Reduces internal covariate shift (though the exact mechanism is debated) - **Higher Learning Rates:** The normalization bounds activations, preventing explosions - **Regularization Effect:** Batch statistics add noise, acting as mild regularization - **Faster Convergence:** Models train significantly faster with BatchNorm **Variants:** - **Layer Normalization:** Normalizes across features (not batch). Used in Transformers where batch size varies. - **Instance Normalization:** Normalizes per-sample per-channel. Used in style transfer. - **Group Normalization:** Normalizes across groups of channels. Works well with small batch sizes. - **RMSNorm:** Only normalizes by RMS, no mean subtraction. Used in LLaMA and modern LLMs. **Important:** The choice between normalization methods depends on the architecture: - CNNs → Batch Normalization - Transformers → Layer Normalization or RMSNorm - Style Transfer → Instance Normalization - Small batches → Group Normalization

Mathematical Notation

x̂ᵢ = (xᵢ − μ_B) / √(σ²_B + ε)    →    yᵢ = γx̂ᵢ + β

Each input xᵢ is normalized using the batch mean μ_B and variance σ²_B (ε prevents division by zero). Learnable parameters γ (scale) and β (shift) allow the network to undo normalization if needed, preserving representational power.

Visual Explanation (flowchart)

Layer Input → Compute Batch Mean & Variance → Normalize → Scale (γ) & Shift (β) → Activation Function → Next Layer

Real-World Use Cases

  • Google: Original BatchNorm paper from Google enabled training much deeper networks
  • Meta: ResNet architectures with BatchNorm remain backbone of many vision systems
  • NVIDIA: EfficientNet uses BatchNorm throughout for efficient image classification

Related Concepts

  • Neural Network — A computing system inspired by biological neural networks that learns patterns from data through interconnected layers of nodes.
  • Deep Learning — A subset of machine learning using neural networks with many layers to learn hierarchical representations from large datasets.
  • Learning Rate — A hyperparameter that controls how large each parameter update step is during gradient descent optimization.
  • Regularization — Techniques that constrain a model's complexity to prevent overfitting and improve generalization to unseen data.

Further Reading

  • Batch Normalization Paper