Advanced · NLP

Sparse Attention

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. Attention mechanisms attending to only a subset of positions, reducing quadratic complexity.

Technical Definition

Attention mechanisms attending to only a subset of positions, reducing quadratic complexity.

How it works

Standard self-attention is O(n²). Sparse attention restricts which positions each token attends to via local windows, strided patterns, or learned patterns. Longformer combines local and global attention. Flash Attention optimizes memory access patterns instead.

Related Concepts

• Transformer — An architecture that uses self-attention to process sequences in parallel, powering modern language models like GPT and BERT.
• Attention Mechanism — A technique that lets models dynamically focus on the most relevant parts of the input when producing each output element.
• KV-Cache — A memory optimization storing previously computed key-value pairs during autoregressive generation to avoid redundant computation.