Advanced · Fundamentals

Contrastive Learning

Visual diagram · (in preparation) · Math · Worked example · 3 difficulty levels.

TL;DR. A self-supervised technique that learns representations by pulling similar samples together and pushing dissimilar ones apart.

Technical Definition

A self-supervised technique that learns representations by pulling similar samples together and pushing dissimilar ones apart.

How it works

Contrastive learning creates positive pairs (augmented views of the same sample) and treats others as negatives. SimCLR, MoCo, and BYOL are landmark approaches. The InfoNCE loss (used in CLIP) is the standard objective. It powers self-supervised pre-training across modalities.

Mathematical Notation

L = -log(exp(sim(zᵢ, zⱼ)/τ) / Σₖ exp(sim(zᵢ, zₖ)/τ))

Maximizes similarity between positive pair relative to all others. Temperature τ controls distribution sharpness.

Related Concepts

• Embedding — A dense vector representation that captures semantic meaning, mapping discrete items like words into continuous mathematical space.
• Transfer Learning — Leveraging knowledge from a model trained on one task to improve performance on a different but related task.
• Zero-Shot Learning — The ability of a model to perform tasks or classify categories it has never explicitly been trained on.