Beginner · NLP

Context Window

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. The maximum number of tokens a model can process in a single input, determining information capacity.

Technical Definition

The maximum number of tokens a model can process in a single input, determining information capacity.

How it works

Context window defines total tokens (input + output) a model handles. GPT-3 had 4K, GPT-4 has 128K, Claude supports 200K, Gemini 1.5 handles 1M+. Challenges include 'lost in the middle' and KV-cache memory scaling.

Related Concepts

• Large Language Model (LLM) — A massive neural network trained on vast text corpora to understand and generate human language with remarkable fluency.
• Tokenization — The process of breaking text into smaller units (tokens) that language models can process as numerical inputs.
• KV-Cache — A memory optimization storing previously computed key-value pairs during autoregressive generation to avoid redundant computation.
• Sparse Attention — Attention mechanisms attending to only a subset of positions, reducing quadratic complexity.