Intermediate · NLP
Generative pretrained transformer (GPT)
Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.
TL;DR. A type of large language model that generates human-like text by predicting words in a sequence after being trained on vast amounts of text data.
Technical Definition
A type of large language model that generates human-like text by predicting words in a sequence after being trained on vast amounts of text data.
How it works
GPT models are built on the transformer architecture and are pre-trained to predict the next word in a sentence. After pre-training, they can generate coherent and contextually relevant text. They are often fine-tuned to improve specific behaviors like accuracy or conversational abilities.
Related Concepts
- Transformer — An architecture that uses self-attention to process sequences in parallel, powering modern language models like GPT and BERT.
- Large Language Model (LLM) — A massive neural network trained on vast text corpora to understand and generate human language with remarkable fluency.
- Fine-Tuning — Adapting a pre-trained model to a specific task by continuing training on a smaller, task-specific dataset.
- Pretraining — The first, expensive training stage where a model learns general patterns from massive unlabeled data.