Advanced · Computer Vision
Vision Transformer (ViT)
Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.
TL;DR. An architecture that applies the Transformer to images by splitting them into patches and processing them as sequences.
Technical Definition
An architecture that applies the Transformer to images by splitting them into patches and processing them as sequences.
How it works
ViT splits images into fixed-size patches, flattens each into a vector, adds positional embeddings, and processes through Transformer encoder layers. It matches or exceeds CNNs when pre-trained on large datasets. Variants include DeiT, Swin Transformer, and DINO.
Related Concepts
- Transformer — An architecture that uses self-attention to process sequences in parallel, powering modern language models like GPT and BERT.
- Attention Mechanism — A technique that lets models dynamically focus on the most relevant parts of the input when producing each output element.
- Convolutional Neural Network (CNN) — A neural network that uses learnable filters to detect spatial patterns like edges, textures, and objects in images.