Intermediate · NLP
Bidirectional language model
Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.
TL;DR. A language model that considers both preceding and succeeding text to determine the probability of a token.
Technical Definition
A language model that considers both preceding and succeeding text to determine the probability of a token.
How it works
A bidirectional language model analyzes text by looking at words both before and after a target word. This allows it to better understand the meaning and context of a word by using information from the entire sentence, leading to more accurate predictions.
Related Concepts
- BERT — A bidirectional Transformer model pre-trained on masked language modeling, revolutionizing NLP benchmarks across the board.
- Language model — A language model is a statistical tool used to predict the probability of a sequence of words occurring in a natural language.