Intermediate · Fundamentals

Optimization

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. The mathematical process of finding parameter values that minimize a loss function.

Technical Definition

The mathematical process of finding parameter values that minimize a loss function.

How it works

Neural network training is a high-dimensional non-convex optimization problem. Algorithms like SGD, Adam, AdamW, and Lion navigate this landscape using gradients. Good optimization requires not just the right algorithm but careful learning-rate schedules, weight initialization, and regularization.

Related Concepts

• Gradient Descent — An optimization algorithm that iteratively adjusts model parameters by moving in the direction of steepest decrease of the loss function.
• Loss Function — A mathematical function that measures how far the model's predictions are from the actual values, guiding the learning process.
• Optimizer — An algorithm that updates model weights during training to minimize the loss function, with strategies beyond basic gradient descent.