Intermediate · Reinforcement Learning

Reward Model

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. A specialized AI model, often used in RLHF, that learns to predict human preferences or scores for different LLM outputs.

Technical Definition

A specialized AI model, often used in RLHF, that learns to predict human preferences or scores for different LLM outputs.

How it works

The reward model is a cornerstone of alignment techniques like RLHF. Trained on human-labeled preference data (e.g., which model response is 'better'), it generates a scalar reward signal. This signal is then used to fine-tune the primary LLM via reinforcement learning, guiding it to produce outputs that are more aligned with human values and intentions, reducing harmful or unhelpful responses.

Related Concepts

• Fine-Tuning — Adapting a pre-trained model to a specific task by continuing training on a smaller, task-specific dataset.
• RLHF (Reinforcement Learning from Human Feedback) — A technique that aligns LLM outputs with human preferences by training a reward model from human comparisons.
• Feedback loop — A feedback loop occurs when a model's output influences its future input or training data, potentially leading to system drift or reinforcement of biases.
• Model Alignment — The process of training AI models, especially LLMs, to behave in accordance with human values, intentions, and ethical principles.