Intermediate · Fundamentals
AdaGrad
Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.
TL;DR. A gradient descent optimization algorithm that adapts the learning rate for each parameter individually.
Technical Definition
A gradient descent optimization algorithm that adapts the learning rate for each parameter individually.
How it works
AdaGrad is an optimization algorithm used in machine learning that modifies the standard gradient descent approach. It adjusts the learning rate for each parameter based on the historical sum of squared gradients for that parameter. This means parameters that have seen large gradients in the past will have their learning rates decreased, while those with smaller historical gradients will have larger learning rates, adapting the optimization process.
Related Concepts
- Gradient Descent — An optimization algorithm that iteratively adjusts model parameters by moving in the direction of steepest decrease of the loss function.
- Learning Rate — A hyperparameter that controls how large each parameter update step is during gradient descent optimization.
- Machine Learning — A field of AI where systems learn patterns from data instead of following hard-coded rules.