Advanced · Systems

PagedAttention

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. A memory-management technique used in vLLM that stores the KV-cache in non-contiguous fixed-size pages, eliminating fragmentation and enabling sharing.

Technical Definition

A memory-management technique used in vLLM that stores the KV-cache in non-contiguous fixed-size pages, eliminating fragmentation and enabling sharing.

How it works

Inspired by virtual memory in operating systems, PagedAttention divides each sequence's key/value cache into pages of a few dozen tokens managed by a block table. This eliminates internal fragmentation, enables copy-on-write sharing for parallel sampling and beam search, and is the core innovation behind vLLM's high-throughput LLM serving.

Related Concepts

• Inference — Using a trained model to make predictions on new data — the deployment phase of machine learning.
• KV-Cache — A memory optimization storing previously computed key-value pairs during autoregressive generation to avoid redundant computation.
• FlashAttention — An IO-aware exact attention algorithm that tiles Q, K, V into SRAM blocks, cutting memory from O(n²) to O(n) and giving 2–4× speedups.