Intermediate · NLP

Positional Encoding

Visual diagram · (in preparation) · Math · Worked example · 3 difficulty levels.

TL;DR. A mechanism that injects sequence order information into Transformers, which otherwise have no notion of position.

Technical Definition

A mechanism that injects sequence order information into Transformers, which otherwise have no notion of position.

How it works

The original Transformer uses sinusoidal functions. Learned positional embeddings are used in BERT/GPT. RoPE encodes relative positions through rotation matrices. ALiBi adds linear bias to attention scores based on token distance.

Mathematical Notation

PE(pos, 2i) = sin(pos/10000^(2i/d))

Sinusoidal encoding uses different frequencies per dimension, allowing the model to learn relative positions.

Related Concepts

• Transformer — An architecture that uses self-attention to process sequences in parallel, powering modern language models like GPT and BERT.
• Attention Mechanism — A technique that lets models dynamically focus on the most relevant parts of the input when producing each output element.
• Embedding — A dense vector representation that captures semantic meaning, mapping discrete items like words into continuous mathematical space.