Intermediate · Neural Networks

Layer Normalization

Visual diagram · (in preparation) · Math · Worked example · 3 difficulty levels.

TL;DR. Standardizes activations across features within each sample, independent of batch size.

Technical Definition

Standardizes activations across features within each sample, independent of batch size.

How it works

Layer norm computes mean and variance across all features for each sample. Unlike batch norm, it doesn't depend on batch statistics. It's the standard in Transformers. RMSNorm, a simplified variant, is used in LLaMA.

Mathematical Notation

LN(x) = γ · (x − μ) / √(σ² + ε) + β

μ and σ² computed across features for a single sample. γ and β are learnable scale and shift.

Related Concepts

• Neural Network — A computing system inspired by biological neural networks that learns patterns from data through interconnected layers of nodes.
• Transformer — An architecture that uses self-attention to process sequences in parallel, powering modern language models like GPT and BERT.
• Batch Normalization — A technique that normalizes layer inputs across a mini-batch, stabilizing and accelerating deep network training.