Intermediate · Systems

Sharding

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. Splitting a large dataset or model across multiple machines or storage devices.

Technical Definition

Splitting a large dataset or model across multiple machines or storage devices.

How it works

Data sharding partitions training data across worker nodes for parallel ingest. Model sharding splits weights across GPUs for models too large for one device. Modern distributed-training frameworks (FSDP, DeepSpeed, Megatron) combine both with pipeline and tensor parallelism.

Related Concepts

• Model Serving — Hosting a trained model behind an interface so applications can request predictions in real time.
• Distributed Training — Training a model across multiple machines or accelerators in parallel.
• Parallelism (Data / Model) — Strategies for splitting computation across devices to train or serve large models.