Intermediate · Neural Networks

Logits

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. The raw, unnormalized scores a model outputs before they are converted to probabilities.

Technical Definition

The raw, unnormalized scores a model outputs before they are converted to probabilities.

How it works

Applying softmax to logits yields a probability distribution. Working with logits directly avoids numerical issues, supports temperature scaling, and is the natural input to cross-entropy loss. In LLMs, the final layer produces one logit per vocabulary token.

Related Concepts

• Softmax Function — A function that converts a vector of raw scores into a probability distribution where all values sum to one.
• Temperature (Sampling) — A parameter controlling output randomness — lower values are more focused, higher values more creative.
• Sampling — The process of choosing the next token from the probability distribution a model outputs.