Advanced · NLP

KV-Cache

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. A memory optimization storing previously computed key-value pairs during autoregressive generation to avoid redundant computation.

Technical Definition

A memory optimization storing previously computed key-value pairs during autoregressive generation to avoid redundant computation.

How it works

Without caching, autoregressive generation recomputes all K/V projections at every step (O(n²) total). KV-cache stores them, reducing per-token cost to O(n). The tradeoff is memory: cache grows linearly with sequence length.

Related Concepts

• Transformer — An architecture that uses self-attention to process sequences in parallel, powering modern language models like GPT and BERT.
• Attention Mechanism — A technique that lets models dynamically focus on the most relevant parts of the input when producing each output element.
• Inference — Using a trained model to make predictions on new data — the deployment phase of machine learning.