Intermediate · NLP

Decoder

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. A model component that generates output sequences, typically one token at a time.

Technical Definition

A model component that generates output sequences, typically one token at a time.

How it works

The decoder of a Transformer uses masked self-attention plus cross-attention over the encoder output to generate translations, summaries, or completions. GPT-style models are decoder-only and generate text autoregressively. In autoencoders, the decoder reconstructs the input from the latent code.

Related Concepts

• Transformer — An architecture that uses self-attention to process sequences in parallel, powering modern language models like GPT and BERT.
• Large Language Model (LLM) — A massive neural network trained on vast text corpora to understand and generate human language with remarkable fluency.
• Encoder — A model component that compresses input data into a dense, information-rich representation.