Advanced · Generative AI
RLHF (Reinforcement Learning from Human Feedback)
Visual diagram · Math · (in preparation) · Worked example · 3 difficulty levels.
TL;DR. A technique that aligns LLM outputs with human preferences by training a reward model from human comparisons.
Technical Definition
A technique that aligns LLM outputs with human preferences by training a reward model from human comparisons.
How it works
RLHF is a three-step process: supervised fine-tuning on demonstrations, training a reward model from human rankings, then optimizing the LLM with PPO against that reward model. It was key to ChatGPT's quality. DPO is a simpler alternative that skips the reward model.
Visual Explanation (flowchart)
Pre-trained LLM → SFT → Generate responses → Human rankings → Train Reward Model → PPO optimization → Aligned Model
Related Concepts
- Large Language Model (LLM) — A massive neural network trained on vast text corpora to understand and generate human language with remarkable fluency.
- Reinforcement Learning — A paradigm where an agent learns to make decisions by receiving rewards or penalties from its environment through trial and error.
- Fine-Tuning — Adapting a pre-trained model to a specific task by continuing training on a smaller, task-specific dataset.
- Policy Gradient — RL algorithms that directly optimize the policy function by gradient ascent on expected reward.