Beginner · NLP

Tokenization

Visual diagram · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. The process of breaking text into smaller units (tokens) that language models can process as numerical inputs.

Technical Definition

The process of breaking text into smaller units (tokens) that language models can process as numerical inputs.

How it works

Tokenization converts raw text into a sequence of tokens — the atomic units a language model operates on. Early approaches used word-level tokenization, but modern models use subword methods like Byte Pair Encoding (BPE) or SentencePiece. These balance vocabulary size with the ability to handle rare and unseen words by breaking them into common subword pieces. For example, 'unhappiness' might become ['un', 'happiness'] or ['un', 'happ', 'iness']. Each token maps to an integer ID, which is then converted to an embedding vector.

Visual Explanation (flowchart)

Raw Text → Tokenizer → Token IDs [382, 1205, 94] → Embedding Layer → Vectors → Model

Related Concepts

• Transformer — An architecture that uses self-attention to process sequences in parallel, powering modern language models like GPT and BERT.
• Large Language Model (LLM) — A massive neural network trained on vast text corpora to understand and generate human language with remarkable fluency.
• Embedding — A dense vector representation that captures semantic meaning, mapping discrete items like words into continuous mathematical space.