Intermediate · NLP
BERT
Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.
TL;DR. A bidirectional Transformer model pre-trained on masked language modeling, revolutionizing NLP benchmarks across the board.
Technical Definition
A bidirectional Transformer model pre-trained on masked language modeling, revolutionizing NLP benchmarks across the board.
How it works
BERT reads text in both directions simultaneously. During pre-training, random tokens are masked and the model learns to predict them using surrounding context. BERT's representations can be fine-tuned for classification, question answering, and named entity recognition with minimal task-specific architecture.
Related Concepts
- Transformer — An architecture that uses self-attention to process sequences in parallel, powering modern language models like GPT and BERT.
- Large Language Model (LLM) — A massive neural network trained on vast text corpora to understand and generate human language with remarkable fluency.
- Fine-Tuning — Adapting a pre-trained model to a specific task by continuing training on a smaller, task-specific dataset.
- Tokenization — The process of breaking text into smaller units (tokens) that language models can process as numerical inputs.