Home › Glossary › NLP › Transformer

Intermediate · NLP

Transformer

Visual diagram · Math · Worked example · 3 difficulty levels.

TL;DR. Transformers are the architecture behind ChatGPT, BERT, and most modern AI. They process all words in a sentence simultaneously using 'attention' to understand context.

Intuitive Explanation

Imagine reading a book where every word can instantly see every other word on the page and decide which ones are most relevant to its meaning. Unlike reading word-by-word (like older RNNs), a Transformer reads the whole page at once, with each word asking 'Who else on this page helps explain me?'

Technical Definition

A Transformer is a sequence-to-sequence neural architecture that replaces recurrence with multi-head self-attention mechanisms. It processes input sequences in parallel by computing scaled dot-product attention over learned Query, Key, and Value projections, combined with positional encodings, layer normalization, residual connections, and position-wise feed-forward networks.

How it works

The Transformer architecture, introduced in "Attention Is All You Need" (Vaswani et al., 2017), consists of two main components: **Encoder Stack (for understanding):** Each encoder layer has two sub-layers: (1) Multi-Head Self-Attention, and (2) Position-wise Feed-Forward Network. Each sub-layer has a residual connection and layer normalization. **Decoder Stack (for generation):** Similar to the encoder but adds a cross-attention layer that attends to the encoder's output. The self-attention in the decoder is masked to prevent attending to future positions. **Self-Attention Mechanism:** 1. Input embeddings are projected into Query (Q), Key (K), and Value (V) matrices 2. Attention scores: score = QKᵀ / √dₖ 3. Softmax normalizes scores into attention weights 4. Output = attention weights × V **Multi-Head Attention:** Instead of one attention function, the model runs h parallel attention heads with different learned projections, then concatenates and projects the results. This lets the model attend to information from different representation subspaces. **Positional Encoding:** Since Transformers process all positions in parallel, they need positional information injected via sinusoidal functions or learned embeddings. **Why Transformers Won:** - Parallelizable (unlike sequential RNNs) - Capture long-range dependencies effectively - Scale extremely well with compute and data - Foundation for all major language models (GPT, BERT, T5, LLaMA)

Mathematical Notation

Attention(Q, K, V) = softmax(QKᵀ / √dₖ) · V

Queries (Q), Keys (K), and Values (V) are linear projections of the input. The dot product QKᵀ measures similarity between tokens. Dividing by √dₖ prevents gradients from vanishing. Softmax converts scores to probabilities, which weight the Values.

Visual Explanation (flowchart)

Input Embedding → Positional Encoding → [Multi-Head Attention → Add & Norm → Feed Forward → Add & Norm] × N → Output

Real-World Use Cases

  • OpenAI: GPT-4 uses a decoder-only Transformer for text generation and reasoning
  • Google: BERT and T5 Transformers power search understanding and translation
  • Meta: LLaMA open-source Transformer models for research and applications
  • Stability AI: Stable Diffusion uses Transformer components for image generation

Related Concepts

  • Large Language Model (LLM) — A massive neural network trained on vast text corpora to understand and generate human language with remarkable fluency.
  • Attention Mechanism — A technique that lets models dynamically focus on the most relevant parts of the input when producing each output element.
  • Embedding — A dense vector representation that captures semantic meaning, mapping discrete items like words into continuous mathematical space.

Further Reading

  • Attention Is All You Need (Paper)
  • The Illustrated Transformer