Advanced · Systems

Distributed Training

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. Training a model across multiple machines or accelerators in parallel.

Technical Definition

Training a model across multiple machines or accelerators in parallel.

How it works

Modern foundation models are trained on hundreds to thousands of GPUs/TPUs using data parallelism (split batches), tensor parallelism (split matrices), pipeline parallelism (split layers), and ZeRO/FSDP for sharded optimizer state. Coordinating gradients across nodes is the central engineering challenge.

Related Concepts

• Sharding — Splitting a large dataset or model across multiple machines or storage devices.
• Parallelism (Data / Model) — Strategies for splitting computation across devices to train or serve large models.
• GPU — Graphics Processing Unit — a massively parallel processor that powers most modern AI workloads.