Beginner · NLP

Token

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. The atomic unit of text a language model processes — typically a word piece or subword.

Technical Definition

The atomic unit of text a language model processes — typically a word piece or subword.

How it works

Tokens are not always whole words. 'Understanding' might split into ['Under', 'standing']; emojis and rare languages can take many tokens. Models have token limits (context windows), and API pricing is usually per token. Roughly 1 token ≈ 0.75 English words.

Related Concepts

• Large Language Model (LLM) — A massive neural network trained on vast text corpora to understand and generate human language with remarkable fluency.
• Tokenization — The process of breaking text into smaller units (tokens) that language models can process as numerical inputs.
• Context Window — The maximum number of tokens a model can process in a single input, determining information capacity.