Intermediate · Generative AI

Pre-training

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. The initial phase of training a large model on a massive dataset, often in an unsupervised manner, to learn general representations.

Technical Definition

The initial phase of training a large model on a massive dataset, often in an unsupervised manner, to learn general representations.

How it works

Pre-training refers to the process of training a large model on a vast dataset, typically with a self-supervised or unsupervised learning objective. The goal is for the model to learn general features and representations from the data. These pre-trained models can then be fine-tuned for specific downstream tasks, leveraging the knowledge acquired during pre-training.

Related Concepts

• Large Language Model (LLM) — A massive neural network trained on vast text corpora to understand and generate human language with remarkable fluency.
• Fine-Tuning — Adapting a pre-trained model to a specific task by continuing training on a smaller, task-specific dataset.
• Transfer Learning — Leveraging knowledge from a model trained on one task to improve performance on a different but related task.