Intermediate · Systems

Batching

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. Grouping multiple inference requests so they run together on the GPU.

Technical Definition

Grouping multiple inference requests so they run together on the GPU.

How it works

Static batching collects N requests, then runs; continuous (in-flight) batching slots new requests into ongoing batches as others finish — crucial for LLMs where outputs have variable length. Batching can boost throughput 5–20× with modest latency cost.

Related Concepts

• Model Serving — Hosting a trained model behind an interface so applications can request predictions in real time.
• Latency — The time between sending a request and receiving the first (or final) response.
• Throughput — The number of requests or tokens a system can process per unit of time.