Beginner · Fundamentals

Mini-batch

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. A small subset of the training data used to compute one gradient update.

Technical Definition

A small subset of the training data used to compute one gradient update.

How it works

Mini-batch gradient descent is the dominant training strategy. It strikes a balance between full-batch (accurate but slow) and stochastic (noisy single-sample) updates. Typical mini-batch sizes range from 16 to a few thousand depending on hardware and model size.

Related Concepts

• Gradient Descent — An optimization algorithm that iteratively adjusts model parameters by moving in the direction of steepest decrease of the loss function.
• Training — The process of adjusting a model's parameters so it learns patterns from labeled or unlabeled data.
• Batch — A group of training examples processed together in a single forward and backward pass.