Beginner · Generative AI

Nucleus Sampling (top-p)

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. A decoding method that samples from the smallest set of tokens whose cumulative probability exceeds p, balancing diversity and coherence.

Technical Definition

A decoding method that samples from the smallest set of tokens whose cumulative probability exceeds p, balancing diversity and coherence.

How it works

Introduced by Holtzman et al. (2019), nucleus sampling truncates the distribution to the smallest 'nucleus' covering probability mass p (typically 0.9–0.95), renormalizes, and samples. Unlike top-k it adapts the candidate set to the distribution's shape — narrow when the model is confident, wide when uncertain — yielding more natural text than greedy or pure sampling.

Related Concepts

• Temperature (Sampling) — A parameter controlling output randomness — lower values are more focused, higher values more creative.