Intermediate · Systems

Load Balancing

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. Distributing incoming requests across multiple servers to maximize utilization and reliability.

Technical Definition

Distributing incoming requests across multiple servers to maximize utilization and reliability.

How it works

Load balancers route traffic by round-robin, least-connections, latency, or model-aware policies (e.g., session affinity for KV-cache reuse). They are the front door of any scalable inference system.

Related Concepts

• Model Serving — Hosting a trained model behind an interface so applications can request predictions in real time.
• Throughput — The number of requests or tokens a system can process per unit of time.
• Scalability — A system's ability to handle increasing load by adding resources.