Advanced · Neural Networks

Grouped-Query Attention (GQA)

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. An attention variant where multiple query heads share each key/value head, shrinking the KV-cache while keeping near-multi-head quality.

Technical Definition

An attention variant where multiple query heads share each key/value head, shrinking the KV-cache while keeping near-multi-head quality.

How it works

Standard multi-head attention has one K and V per query head; multi-query attention (MQA) shares a single K/V across all heads. GQA interpolates: G groups of query heads each share one K/V head. This cuts KV-cache memory and bandwidth roughly H/G times — a major inference speed-up at long context — with negligible quality loss. Used in LLaMA-2/3, Mistral, and most recent open LLMs.

Related Concepts

• Transformer — An architecture that uses self-attention to process sequences in parallel, powering modern language models like GPT and BERT.
• Attention Mechanism — A technique that lets models dynamically focus on the most relevant parts of the input when producing each output element.
• KV-Cache — A memory optimization storing previously computed key-value pairs during autoregressive generation to avoid redundant computation.