Advanced · NLP

Self-Attention

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. A mechanism where every token in a sequence attends to every other token to compute context-aware representations.

Technical Definition

A mechanism where every token in a sequence attends to every other token to compute context-aware representations.

How it works

Self-attention is the core operation of the Transformer. For each token, it computes Query, Key, and Value vectors, then uses scaled dot-product attention to produce a weighted sum of all Values. Multi-head self-attention runs this in parallel across several subspaces, letting the model capture different kinds of relationships simultaneously.

Related Concepts

• Transformer — An architecture that uses self-attention to process sequences in parallel, powering modern language models like GPT and BERT.
• Attention Mechanism — A technique that lets models dynamically focus on the most relevant parts of the input when producing each output element.
• Attention Head — One of multiple parallel attention computations, each learning to focus on different types of relationships in the data.