Beginner · Fundamentals
Convergence
Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.
TL;DR. The point in model training where improvements in performance (like reduced loss) become negligible with further iterations.
Technical Definition
The point in model training where improvements in performance (like reduced loss) become negligible with further iterations.
How it works
Convergence occurs when a model's training process reaches a stable state, meaning that performance metrics, particularly the loss value, change very little or stop changing altogether with each subsequent training iteration. Reaching convergence indicates that further training is unlikely to yield significant improvements in the model's accuracy. However, a prolonged period of minimal change in loss can sometimes give a false impression of convergence.
Related Concepts
- Loss Function — A mathematical function that measures how far the model's predictions are from the actual values, guiding the learning process.
- Iteration — One update step of the model's parameters, processing a single batch.
- Early stopping — A regularization technique used in iterative training to stop model learning when performance on a validation set starts to degrade.