Home › Glossary › Reinforcement Learning › GRPO (Group Relative Policy Optimization)

Advanced · Reinforcement Learning

GRPO (Group Relative Policy Optimization)

Visual diagram · (in preparation) · Math · (in preparation) · Worked example · 3 difficulty levels.

TL;DR. A PPO variant from DeepSeek that drops the value network and estimates advantages from the relative reward of a group of sampled completions.

Technical Definition

A PPO variant from DeepSeek that drops the value network and estimates advantages from the relative reward of a group of sampled completions.

How it works

GRPO samples G responses per prompt, scores each with a reward model or verifier, and computes each sample's advantage as its normalized rank within the group. Removing the critic saves memory and avoids value-function bias. GRPO powered DeepSeek-R1's emergent chain-of-thought training and is widely used for reasoning RL on math and code benchmarks.

Related Concepts

  • Reinforcement Learning — A paradigm where an agent learns to make decisions by receiving rewards or penalties from its environment through trial and error.
  • RLHF (Reinforcement Learning from Human Feedback) — A technique that aligns LLM outputs with human preferences by training a reward model from human comparisons.
  • Direct Preference Optimization (DPO) — A simpler RLHF alternative that fine-tunes an LLM directly on preference pairs using a closed-form loss, with no reward model or RL loop.