Advanced · Neural Networks

RMSNorm

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. A normalization layer that rescales activations by their root-mean-square, dropping the mean-centering step of LayerNorm for ~10% speed-up.

Technical Definition

A normalization layer that rescales activations by their root-mean-square, dropping the mean-centering step of LayerNorm for ~10% speed-up.

How it works

RMSNorm (Zhang & Sennrich, 2019) normalizes a vector x by x / sqrt(mean(x²) + ε) and applies a learned per-channel gain. Compared with LayerNorm it removes the bias and the mean subtraction, simplifying the kernel and reducing memory traffic. Empirically it matches LayerNorm's stability and is the default norm in LLaMA, T5, and most modern Transformers.

Related Concepts

• Transformer — An architecture that uses self-attention to process sequences in parallel, powering modern language models like GPT and BERT.
• Batch Normalization — A technique that normalizes layer inputs across a mini-batch, stabilizing and accelerating deep network training.
• Layer Normalization — Standardizes activations across features within each sample, independent of batch size.