Beginner · Systems

Latency

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. The time between sending a request and receiving the first (or final) response.

Technical Definition

The time between sending a request and receiving the first (or final) response.

How it works

For LLMs, time-to-first-token and tokens-per-second matter as much as total latency. Reducing latency uses techniques like quantization, KV caching, speculative decoding, and edge deployment. Latency is felt by every user; small wins compound across millions of requests.

Related Concepts

• Model Serving — Hosting a trained model behind an interface so applications can request predictions in real time.
• Throughput — The number of requests or tokens a system can process per unit of time.
• Scalability — A system's ability to handle increasing load by adding resources.