Advanced · Neural Networks

RoPE (Rotary Position Embedding)

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. A positional encoding that rotates query and key vectors by an angle proportional to position, giving relative-position behavior with no extra parameters.

Technical Definition

A positional encoding that rotates query and key vectors by an angle proportional to position, giving relative-position behavior with no extra parameters.

How it works

RoPE (Su et al., 2021) multiplies pairs of dimensions in Q and K by a 2D rotation matrix whose angle depends on the token's position and a base frequency. The dot product QKᵀ then depends only on the relative offset, giving translation equivariance for free. RoPE is the de-facto positional scheme in LLaMA, Mistral, Qwen, and most modern LLMs, and underpins context-extension methods like YaRN and NTK-aware scaling.

Related Concepts

• Transformer — An architecture that uses self-attention to process sequences in parallel, powering modern language models like GPT and BERT.
• Attention Mechanism — A technique that lets models dynamically focus on the most relevant parts of the input when producing each output element.
• Positional Encoding — A mechanism that injects sequence order information into Transformers, which otherwise have no notion of position.
• Long Context — The ability of an LLM to process and maintain coherence over very long input sequences or conversations, beyond typical token limits.