Advanced · Generative AI

Direct Preference Optimization (DPO)

Visual diagram · (in preparation) · Math · Worked example · 3 difficulty levels.

TL;DR. A simpler RLHF alternative that fine-tunes an LLM directly on preference pairs using a closed-form loss, with no reward model or RL loop.

Technical Definition

A simpler RLHF alternative that fine-tunes an LLM directly on preference pairs using a closed-form loss, with no reward model or RL loop.

How it works

DPO (Rafailov et al., 2023) reparameterizes the RLHF objective so the optimal policy can be derived analytically from the reward. Training reduces to a binary classification loss over chosen vs. rejected completions, comparing log-probabilities under the policy and a frozen reference. It is far more stable and cheaper than PPO-based RLHF and is now the default alignment recipe for many open-weight models.

Mathematical Notation

L_DPO = -log σ(β log π(y_w|x)/π_ref(y_w|x) - β log π(y_l|x)/π_ref(y_l|x))

Related Concepts

• Fine-Tuning — Adapting a pre-trained model to a specific task by continuing training on a smaller, task-specific dataset.
• RLHF (Reinforcement Learning from Human Feedback) — A technique that aligns LLM outputs with human preferences by training a reward model from human comparisons.