Intermediate · Fundamentals

Self-Supervised Learning

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. A training paradigm that generates supervisory signals from the data itself, eliminating the need for human labels.

Technical Definition

A training paradigm that generates supervisory signals from the data itself, eliminating the need for human labels.

How it works

Self-supervised learning creates objectives from unlabeled data. Masked language modeling powers BERT, next-token prediction powers GPT. In vision, contrastive methods and masked image modeling learn features without labels. It's the dominant pre-training paradigm.

Related Concepts

• Transfer Learning — Leveraging knowledge from a model trained on one task to improve performance on a different but related task.
• BERT — A bidirectional Transformer model pre-trained on masked language modeling, revolutionizing NLP benchmarks across the board.
• Contrastive Learning — A self-supervised technique that learns representations by pulling similar samples together and pushing dissimilar ones apart.