Intermediate · NLP

Attention Head

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. One of multiple parallel attention computations, each learning to focus on different types of relationships in the data.

Technical Definition

One of multiple parallel attention computations, each learning to focus on different types of relationships in the data.

How it works

In multi-head attention, inputs are projected into multiple independent Q, K, V sets. Different heads learn different relationship types: syntactic, positional, semantic. Outputs are concatenated and projected. GPT-4 has 96+ heads per layer.

Related Concepts

• Transformer — An architecture that uses self-attention to process sequences in parallel, powering modern language models like GPT and BERT.
• Large Language Model (LLM) — A massive neural network trained on vast text corpora to understand and generate human language with remarkable fluency.
• Attention Mechanism — A technique that lets models dynamically focus on the most relevant parts of the input when producing each output element.