Intermediate · Neural Networks

Autoregressive Model

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. A statistical model that predicts future values based on past values, where each prediction contributes to the context for the next.

Technical Definition

A statistical model that predicts future values based on past values, where each prediction contributes to the context for the next.

How it works

In the context of LLMs, autoregressive models generate text token by token, sequentially building upon the previously generated output. This means that each new word or element is conditioned on all preceding elements in the sequence. Decoder-only transformers are prime examples of autoregressive models, excelling at tasks like text completion, translation, and summarization.

Related Concepts

• Decoder-only Transformer — A type of transformer model that consists solely of a stack of decoder layers, primarily used for generative tasks like text generation.