Beginner · Fundamentals
Embedding
Visual diagram · (in preparation) · Math · Worked example · 3 difficulty levels.
TL;DR. Embeddings convert words, images, or any data into lists of numbers (vectors) where similar items are close together in mathematical space.
Intuitive Explanation
Imagine a map where cities are placed by similarity rather than geography — cities with similar cultures, climates, and languages cluster together. Embeddings create this kind of 'meaning map' for words, where 'king' and 'queen' are neighbors, and 'king - man + woman ≈ queen.'
Technical Definition
An embedding is a learned mapping from a discrete, high-dimensional space (vocabulary, item catalog) to a continuous, lower-dimensional vector space where geometric relationships encode semantic similarity. Formally, E: V → ℝᵈ where V is the vocabulary and d << |V|.
How it works
Embeddings are the bridge between human concepts and mathematical computation. **Why Embeddings Matter:** Computers process numbers, not words. One-hot encoding (a vector of zeros with a single 1) treats every word as equally different from every other word. Embeddings solve this by learning dense representations where semantic relationships are geometric relationships. **Key Properties:** - **Dimensionality Reduction:** |V| = 50,000+ vocabulary → d = 256-4096 dimensions - **Semantic Similarity:** cos(embedding("cat"), embedding("dog")) > cos(embedding("cat"), embedding("car")) - **Compositionality:** Vector arithmetic captures relationships: king - man + woman ≈ queen - **Transfer Learning:** Pre-trained embeddings capture general language knowledge **Types:** - **Word Embeddings:** Word2Vec, GloVe — fixed vectors per word, no context - **Contextual Embeddings:** BERT, GPT — different vector for "bank" in "river bank" vs "bank account" - **Sentence Embeddings:** all-MiniLM, e5 — represent entire sentences as vectors - **Multimodal Embeddings:** CLIP — shared space for text and images **Applications:** - Semantic search (find documents by meaning, not keywords) - Recommendation systems (embed users and items in same space) - Clustering and visualization (t-SNE/UMAP of embeddings) - RAG (retrieve relevant context for LLMs)
Mathematical Notation
e = W · one_hot(token) where W ∈ ℝ^{d×V}The embedding matrix W has d dimensions (typically 256–4096) and V columns (vocabulary size). Multiplying by a one-hot vector selects the column corresponding to that token.
Real-World Use Cases
- Spotify: Embeds songs and users in the same space for music recommendations
- Google: BERT embeddings power semantic search and featured snippets
- Pinecone: Vector database built specifically for storing and searching embeddings
- OpenAI: text-embedding-ada-002 provides embeddings for search, clustering, and RAG
Related Concepts
- Transformer — An architecture that uses self-attention to process sequences in parallel, powering modern language models like GPT and BERT.
- Large Language Model (LLM) — A massive neural network trained on vast text corpora to understand and generate human language with remarkable fluency.
- Attention Mechanism — A technique that lets models dynamically focus on the most relevant parts of the input when producing each output element.