Intermediate · NLP
Causal language model
Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.
TL;DR. A language model that predicts the next token based only on the preceding tokens, meaning it processes text in one direction.
Technical Definition
A language model that predicts the next token based only on the preceding tokens, meaning it processes text in one direction.
How it works
A causal language model, often referred to as a unidirectional language model, is designed to predict the next word in a sequence given all the previous words. This directional approach is fundamental for tasks like text generation, where the model builds upon what it has already written. It contrasts with models that can process text bidirectionally.
Related Concepts
- Bidirectional language model — A language model that considers both preceding and succeeding text to determine the probability of a token.