Home › Glossary › Fundamentals › Loss Function

Beginner · Fundamentals

Loss Function

Visual diagram · (in preparation) · Math · Worked example · 3 difficulty levels.

TL;DR. A loss function measures how wrong a model's predictions are. It's the score that training tries to minimize — lower loss means better predictions.

Intuitive Explanation

A loss function is like a report card for an AI model. It gives a score for how badly the model performed on each test question. The student (model) then studies (adjusts weights) specifically to improve on the areas where the score was worst.

Technical Definition

A loss function L: ℝⁿ × ℝⁿ → ℝ maps a model's predictions ŷ and ground truth labels y to a non-negative scalar quantifying prediction error. It defines the objective surface that gradient-based optimization traverses during training.

How it works

The choice of loss function fundamentally shapes what a model learns. **Common Loss Functions:** **For Regression:** - **MSE (Mean Squared Error):** L = (1/n)Σ(yᵢ - ŷᵢ)². Penalizes large errors heavily. Sensitive to outliers. - **MAE (Mean Absolute Error):** L = (1/n)Σ|yᵢ - ŷᵢ|. More robust to outliers than MSE. - **Huber Loss:** MSE for small errors, MAE for large errors. Best of both worlds. **For Classification:** - **Cross-Entropy:** L = -Σ yᵢ log(ŷᵢ). The standard for classification. Works with softmax outputs. - **Binary Cross-Entropy:** For binary classification. L = -(y log(ŷ) + (1-y)log(1-ŷ)). - **Focal Loss:** Down-weights easy examples to focus on hard ones. Used in object detection. **For Generative Models:** - **KL Divergence:** Measures how one probability distribution differs from another. Used in VAEs. - **Adversarial Loss:** GAN loss based on discriminator confidence. - **Perceptual Loss:** Uses a pre-trained network to compare high-level features. **For Contrastive Learning:** - **InfoNCE / Contrastive Loss:** Pulls similar pairs together, pushes dissimilar pairs apart. - **Triplet Loss:** anchor-positive distance < anchor-negative distance + margin. **Choosing a Loss Function:** The loss function should match your task, data characteristics, and what you care about. Wrong loss = model optimizes for the wrong thing.

Mathematical Notation

MSE = (1/n) Σ(yᵢ − ŷᵢ)²    CE = −Σ yᵢ log(ŷᵢ)

MSE averages the squared differences between actual (y) and predicted (ŷ) values. Cross-Entropy (CE) penalizes confident wrong predictions heavily — predicting 0.01 when the true label is 1 incurs a large loss.

Real-World Use Cases

  • OpenAI: Cross-entropy loss on next-token prediction drives all GPT model training
  • Tesla: Multi-task losses combining detection, segmentation, and depth estimation
  • Google: Contrastive losses power CLIP and multimodal alignment

Related Concepts

  • Neural Network — A computing system inspired by biological neural networks that learns patterns from data through interconnected layers of nodes.
  • Gradient Descent — An optimization algorithm that iteratively adjusts model parameters by moving in the direction of steepest decrease of the loss function.
  • Backpropagation — An algorithm that efficiently computes gradients by propagating errors backward through the network using the chain rule.