Home › Glossary › NLP › Attention Mechanism

Intermediate · NLP

Attention Mechanism

Visual diagram · Math · Worked example · 3 difficulty levels.

TL;DR. Attention lets AI models focus on the most relevant parts of input data, similar to how you focus on specific words when reading a sentence.

Intuitive Explanation

When reading 'The cat sat on the mat because it was tired,' your brain automatically connects 'it' to 'cat' — not 'mat.' Attention does the same thing for AI: it looks at all the words and decides which ones are most related to each other.

Technical Definition

An attention mechanism computes a weighted sum of value vectors, where the weights are derived from a compatibility function between query and key vectors. In scaled dot-product attention: Attention(Q,K,V) = softmax(QKᵀ/√dₖ)V, enabling dynamic, content-dependent information routing.

How it works

Attention is the fundamental innovation behind modern AI's success with sequential data. **Types of Attention:** 1. **Self-Attention:** Each position attends to all positions in the same sequence. Used in Transformer encoders. 2. **Cross-Attention:** Queries come from one sequence, keys/values from another. Used in encoder-decoder models. 3. **Causal (Masked) Attention:** Each position can only attend to previous positions. Used in autoregressive models like GPT. **Multi-Head Attention:** Instead of a single attention function, multi-head attention runs h parallel attention heads with different learned projections: MultiHead(Q,K,V) = Concat(head₁,...,headₕ)Wᴼ Each head can learn to attend to different types of relationships (syntactic, semantic, positional). **Attention Variants:** - **Sparse Attention:** Only attend to nearby positions + strided positions. Reduces O(n²) to O(n√n). - **Linear Attention:** Approximates softmax attention with kernel tricks. O(n) complexity. - **Flash Attention:** Hardware-aware algorithm that computes exact attention with O(n) memory instead of O(n²). - **Grouped Query Attention (GQA):** Shares K,V heads across multiple Q heads. Used in LLaMA 2+ for efficient inference. **Attention as Soft Dictionary Lookup:** Think of attention as a differentiable dictionary: queries are lookup keys, keys are dictionary keys, and values are dictionary values. The attention weights determine how much of each value to retrieve.

Mathematical Notation

αᵢⱼ = exp(eᵢⱼ) / Σₖ exp(eᵢₖ)    where eᵢⱼ = score(sᵢ, hⱼ)

The attention weight αᵢⱼ tells the model how much to focus on input position j when producing output position i. The score function measures compatibility between the decoder state sᵢ and encoder hidden state hⱼ.

Visual Explanation (matrix)

Attention Matrix: rows = output tokens, columns = input tokens. Brighter cells = higher attention weight.

Real-World Use Cases

  • Google: Attention mechanism is core to Google Translate and Search AI
  • OpenAI: GPT's causal self-attention enables autoregressive text generation
  • DeepMind: AlphaFold 2 uses attention to model protein structure relationships

Related Concepts

  • Transformer — An architecture that uses self-attention to process sequences in parallel, powering modern language models like GPT and BERT.
  • Large Language Model (LLM) — A massive neural network trained on vast text corpora to understand and generate human language with remarkable fluency.
  • Embedding — A dense vector representation that captures semantic meaning, mapping discrete items like words into continuous mathematical space.