Beginner · Neural Networks

Softmax Function

Visual diagram · (in preparation) · Math · Worked example · 3 difficulty levels.

TL;DR. A function that converts a vector of raw scores into a probability distribution where all values sum to one.

Technical Definition

A function that converts a vector of raw scores into a probability distribution where all values sum to one.

How it works

Softmax is the standard output activation for multi-class classification. It exponentiates each score, then normalizes by dividing by the sum of all exponentiated scores. This ensures outputs are positive and sum to 1, making them interpretable as probabilities.

Mathematical Notation

softmax(zᵢ) = exp(zᵢ) / Σⱼ exp(zⱼ)

Each raw score zᵢ is exponentiated to ensure positivity, then normalized by the sum of all exponentiated scores.

Related Concepts

• Neural Network — A computing system inspired by biological neural networks that learns patterns from data through interconnected layers of nodes.
• Activation Function — A non-linear function applied to a neuron's output, enabling the network to learn complex, non-linear relationships.
• Loss Function — A mathematical function that measures how far the model's predictions are from the actual values, guiding the learning process.