Intermediate · NLP
Word2Vec
Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.
TL;DR. A pioneering technique that learns dense word embeddings by predicting surrounding words from large text corpora.
Technical Definition
A pioneering technique that learns dense word embeddings by predicting surrounding words from large text corpora.
How it works
Word2Vec learns word representations in two flavors: CBOW predicts a target word from context, Skip-gram predicts context from a target word. The resulting vectors capture semantic relationships — vector('king') - vector('man') + vector('woman') ≈ vector('queen'). It laid the groundwork for all modern embedding approaches.
Related Concepts
- Transformer — An architecture that uses self-attention to process sequences in parallel, powering modern language models like GPT and BERT.
- Embedding — A dense vector representation that captures semantic meaning, mapping discrete items like words into continuous mathematical space.
- Tokenization — The process of breaking text into smaller units (tokens) that language models can process as numerical inputs.