Intermediate · Systems

Scalability

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. A system's ability to handle increasing load by adding resources.

Technical Definition

A system's ability to handle increasing load by adding resources.

How it works

Vertical scaling adds bigger machines; horizontal scaling adds more machines behind a load balancer. ML systems also scale by sharding data, models, or both. True scalability requires designing for stateless services, async work, and graceful degradation under failure.

Related Concepts

• Throughput — The number of requests or tokens a system can process per unit of time.
• Load Balancing — Distributing incoming requests across multiple servers to maximize utilization and reliability.
• Distributed Training — Training a model across multiple machines or accelerators in parallel.