Intermediate · Neural Networks

Decoder-only Transformer

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. A type of transformer model that consists solely of a stack of decoder layers, primarily used for generative tasks like text generation.

Technical Definition

A type of transformer model that consists solely of a stack of decoder layers, primarily used for generative tasks like text generation.

How it works

Decoder-only transformers, like GPT models, are designed to generate sequences autoregressively—predicting the next token based on all previous tokens. They are excellent for tasks like creative writing, summarization, and question answering where the output needs to be generated step-by-step. They do not encode an input sequence into a fixed representation, but rather directly produce outputs.

Related Concepts

• Transformer Architecture — A neural network architecture, predominantly used in NLP, that relies heavily on self-attention mechanisms to process sequential data.
• Autoregressive Model — A statistical model that predicts future values based on past values, where each prediction contributes to the context for the next.