Advanced · Computer Vision
CLIP
Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.
TL;DR. A multi-modal model connecting images and text in a shared embedding space for zero-shot visual classification.
Technical Definition
A multi-modal model connecting images and text in a shared embedding space for zero-shot visual classification.
How it works
CLIP trains vision and text encoders jointly on 400M image-text pairs using contrastive learning. Any text description can serve as a classifier at inference. This enables zero-shot classification without labeled training data for target categories.
Related Concepts
- Embedding — A dense vector representation that captures semantic meaning, mapping discrete items like words into continuous mathematical space.
- Vision Transformer (ViT) — An architecture that applies the Transformer to images by splitting them into patches and processing them as sequences.
- Zero-Shot Learning — The ability of a model to perform tasks or classify categories it has never explicitly been trained on.
- Contrastive Learning — A self-supervised technique that learns representations by pulling similar samples together and pushing dissimilar ones apart.