Intermediate · Generative AI
Retrieval-Augmented Generation (RAG)
Visual diagram · Math · (in preparation) · Worked example · 3 difficulty levels.
TL;DR. RAG (Retrieval-Augmented Generation) gives AI models access to external knowledge by retrieving relevant documents before generating a response, reducing hallucinations.
Intuitive Explanation
Imagine a student taking an open-book exam. Instead of relying only on memory (which might be wrong), they look up relevant pages in their textbook before answering each question. RAG does the same for AI — it retrieves relevant documents and uses them to ground its response in facts.
Technical Definition
Retrieval-Augmented Generation is a framework that combines a neural retriever (typically dense passage retrieval using bi-encoder embeddings) with a generative language model. The retriever finds relevant documents from a knowledge base, and the generator conditions its output on both the query and retrieved context, improving factual accuracy and enabling knowledge updates without retraining.
How it works
RAG is one of the most practical and important patterns in production AI systems. **Why RAG:** - LLMs have knowledge cutoff dates and can hallucinate - Retraining on new data is expensive and slow - RAG provides up-to-date, verifiable, source-cited answers **Architecture:** 1. **Indexing:** Documents are chunked, embedded, and stored in a vector database 2. **Retrieval:** User query is embedded, and top-k similar chunks are retrieved via ANN search 3. **Augmentation:** Retrieved chunks are injected into the LLM prompt as context 4. **Generation:** The LLM generates a response grounded in the retrieved context **Advanced Techniques:** - **Hybrid Search:** Combine dense retrieval (semantic) with sparse retrieval (BM25/keyword) for better coverage - **Re-ranking:** Use a cross-encoder to re-rank retrieved documents for relevance - **Query Decomposition:** Break complex queries into sub-queries for multi-hop retrieval - **Chunk Optimization:** Tune chunk size, overlap, and metadata for retrieval quality - **Self-RAG:** The model decides when to retrieve and evaluates its own citations **Evaluation:** - **Retrieval metrics:** Recall@k, MRR, NDCG - **Generation metrics:** Faithfulness (does the answer match sources?), relevance, completeness - **End-to-end:** Human evaluation of answer quality and citation accuracy
Visual Explanation (flowchart)
User Query → Embed Query → Vector Search (Knowledge Base) → Top-K Documents → [Query + Retrieved Context] → LLM → Grounded Answer
Real-World Use Cases
- Perplexity AI: Search engine built entirely on RAG, providing cited answers from web sources
- Microsoft: Copilot uses RAG to ground responses in your documents, emails, and files
- Notion: Notion AI uses RAG to answer questions about your workspace content
- Legal/Medical AI: RAG enables AI assistants to cite specific statutes, papers, or guidelines
Related Concepts
- Transformer — An architecture that uses self-attention to process sequences in parallel, powering modern language models like GPT and BERT.
- Large Language Model (LLM) — A massive neural network trained on vast text corpora to understand and generate human language with remarkable fluency.
- Embedding — A dense vector representation that captures semantic meaning, mapping discrete items like words into continuous mathematical space.
- Vector Database — A specialized database optimized for storing, indexing, and querying high-dimensional embedding vectors using similarity search.