Intermediate · Systems
Gradient accumulation
Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.
TL;DR. A technique to simulate larger batch sizes during training by accumulating gradients over multiple mini-batches before performing a single parameter update.
Technical Definition
A technique to simulate larger batch sizes during training by accumulating gradients over multiple mini-batches before performing a single parameter update.
How it works
Gradient accumulation is a method used in deep learning to train models with effectively larger batch sizes than what fits into GPU memory. Instead of updating model weights after each small batch, gradients are computed for several mini-batches and accumulated. A single weight update is then performed using the summed gradients from these mini-batches, mimicking the effect of a larger batch size and potentially improving model stability and performance.
Related Concepts
- Backpropagation — An algorithm that efficiently computes gradients by propagating errors backward through the network using the chain rule.
- Deep Learning — A subset of machine learning using neural networks with many layers to learn hierarchical representations from large datasets.
- Optimization — The mathematical process of finding parameter values that minimize a loss function.
- Batch size — The number of training examples processed in one iteration of a machine learning algorithm's update cycle.