Intermediate · Generative AI

Pretraining

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. The first, expensive training stage where a model learns general patterns from massive unlabeled data.

Technical Definition

The first, expensive training stage where a model learns general patterns from massive unlabeled data.

How it works

Pretraining for LLMs typically uses next-token prediction on trillions of tokens of web text, code, and books. The result is a foundation model with broad knowledge but no specific task behavior. Pretraining can cost millions of dollars in compute; downstream fine-tuning is comparatively cheap.

Related Concepts

• Fine-Tuning — Adapting a pre-trained model to a specific task by continuing training on a smaller, task-specific dataset.
• Self-Supervised Learning — A training paradigm that generates supervisory signals from the data itself, eliminating the need for human labels.