Intermediate · NLP

Perplexity

Visual diagram · (in preparation) · Math · Worked example · 3 difficulty levels.

TL;DR. A metric measuring how well a language model predicts text — lower means less 'surprised' by the data.

Technical Definition

A metric measuring how well a language model predicts text — lower means less 'surprised' by the data.

How it works

Perplexity is the exponential of average cross-entropy loss. A perplexity of 10 means the model is as uncertain as choosing among 10 options. Lower is better. Typical values for good LLMs are 5-20 on English text.

Mathematical Notation

PPL = exp(-(1/N) Σ log P(wᵢ | w₁...wᵢ₋₁))

Exponentiates the negative average log-likelihood. Perfect prediction approaches PPL=1.

Related Concepts

• Large Language Model (LLM) — A massive neural network trained on vast text corpora to understand and generate human language with remarkable fluency.
• Loss Function — A mathematical function that measures how far the model's predictions are from the actual values, guiding the learning process.
• Tokenization — The process of breaking text into smaller units (tokens) that language models can process as numerical inputs.