Advanced · Systems

Speculative Decoding

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. An inference technique where a small draft model proposes several tokens which a large target model verifies in parallel, accelerating generation.

Technical Definition

An inference technique where a small draft model proposes several tokens which a large target model verifies in parallel, accelerating generation.

How it works

A small, fast draft model autoregressively generates K candidate tokens. The large target model then evaluates the draft in a single parallel forward pass and accepts the longest prefix that matches its own distribution under rejection sampling. Accepted tokens are emitted; the rest are resampled. Output is provably identical to the target model's, but 2–3× faster on typical workloads.

Related Concepts

• Large Language Model (LLM) — A massive neural network trained on vast text corpora to understand and generate human language with remarkable fluency.
• Inference — Using a trained model to make predictions on new data — the deployment phase of machine learning.
• KV-Cache — A memory optimization storing previously computed key-value pairs during autoregressive generation to avoid redundant computation.