Intermediate · Generative AI

Top-k Sampling

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. A decoding strategy that restricts the next-token choice to the k most likely tokens.

Technical Definition

A decoding strategy that restricts the next-token choice to the k most likely tokens.

How it works

After computing the model's probability distribution, top-k zeros out everything outside the top k candidates and renormalizes, then samples from this truncated distribution. Small k (e.g. 10) gives focused outputs; large k approaches unrestricted sampling. Often combined with temperature.

Related Concepts

• Temperature (Sampling) — A parameter controlling output randomness — lower values are more focused, higher values more creative.
• Sampling — The process of choosing the next token from the probability distribution a model outputs.
• Top-p (Nucleus) Sampling — A decoding strategy that samples from the smallest set of tokens whose cumulative probability exceeds p.