Intermediate · Systems

Throughput

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. The number of requests or tokens a system can process per unit of time.

Technical Definition

The number of requests or tokens a system can process per unit of time.

How it works

Latency cares about a single request; throughput cares about the fleet. Batching, parallelism, and efficient kernels increase throughput, often at small latency cost. Maximizing tokens-per-GPU-second is the dominant economic metric for LLM serving.

Related Concepts

• Latency — The time between sending a request and receiving the first (or final) response.
• Scalability — A system's ability to handle increasing load by adding resources.
• Batching — Grouping multiple inference requests so they run together on the GPU.