Intermediate · Fundamentals

Gradient

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. The vector of partial derivatives of the loss with respect to each model parameter.

Technical Definition

The vector of partial derivatives of the loss with respect to each model parameter.

How it works

The gradient points in the direction of steepest increase of the loss; gradient descent moves in the opposite direction to reduce it. Gradients are computed efficiently via backpropagation using the chain rule, allowing networks with billions of parameters to be trained.

Related Concepts

• Gradient Descent — An optimization algorithm that iteratively adjusts model parameters by moving in the direction of steepest decrease of the loss function.
• Backpropagation — An algorithm that efficiently computes gradients by propagating errors backward through the network using the chain rule.
• Loss Function — A mathematical function that measures how far the model's predictions are from the actual values, guiding the learning process.