Advanced · Systems

Parallelism (Data / Model)

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. Strategies for splitting computation across devices to train or serve large models.

Technical Definition

Strategies for splitting computation across devices to train or serve large models.

How it works

Data parallelism replicates the model and splits the batch; model parallelism splits the model itself across devices (tensor or pipeline). Frontier training runs combine all three. Choosing the mix depends on model size, batch size, and interconnect bandwidth.

Related Concepts

• Sharding — Splitting a large dataset or model across multiple machines or storage devices.
• Distributed Training — Training a model across multiple machines or accelerators in parallel.
• GPU — Graphics Processing Unit — a massively parallel processor that powers most modern AI workloads.