Intermediate · Generative AI

Sampling

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. The process of choosing the next token from the probability distribution a model outputs.

Technical Definition

The process of choosing the next token from the probability distribution a model outputs.

How it works

Greedy sampling always picks the highest-probability token; it is deterministic but bland. Temperature, top-k, and top-p sampling introduce controlled randomness, producing more diverse and creative outputs. Sampling strategy strongly affects whether a model feels robotic or natural.

Related Concepts

• Temperature (Sampling) — A parameter controlling output randomness — lower values are more focused, higher values more creative.
• Top-k Sampling — A decoding strategy that restricts the next-token choice to the k most likely tokens.
• Top-p (Nucleus) Sampling — A decoding strategy that samples from the smallest set of tokens whose cumulative probability exceeds p.