Intermediate · Generative AI

Top-p (Nucleus) Sampling

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. A decoding strategy that samples from the smallest set of tokens whose cumulative probability exceeds p.

Technical Definition

A decoding strategy that samples from the smallest set of tokens whose cumulative probability exceeds p.

How it works

Unlike top-k's fixed cutoff, nucleus sampling adapts: when the model is confident, only a few tokens make the cut; when uncertain, the pool widens. Typical values are p = 0.9–0.95. It generally produces more coherent text than top-k at the same level of diversity.

Related Concepts

• Temperature (Sampling) — A parameter controlling output randomness — lower values are more focused, higher values more creative.
• Sampling — The process of choosing the next token from the probability distribution a model outputs.
• Top-k Sampling — A decoding strategy that restricts the next-token choice to the k most likely tokens.