Beginner · Systems

Caching

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. Storing the results of expensive computations so they can be reused without recomputing.

Technical Definition

Storing the results of expensive computations so they can be reused without recomputing.

How it works

In LLM serving, KV caching avoids recomputing attention over earlier tokens. In RAG, embedding caches avoid re-embedding queries. Application-level caches (Redis) cache full responses. Caching is one of the highest-ROI performance optimizations available.

Related Concepts

• KV-Cache — A memory optimization storing previously computed key-value pairs during autoregressive generation to avoid redundant computation.
• Latency — The time between sending a request and receiving the first (or final) response.
• Throughput — The number of requests or tokens a system can process per unit of time.