Advanced · Reinforcement Learning

Policy Gradient

Visual diagram · (in preparation) · Math · Worked example · 3 difficulty levels.

TL;DR. RL algorithms that directly optimize the policy function by gradient ascent on expected reward.

Technical Definition

RL algorithms that directly optimize the policy function by gradient ascent on expected reward.

How it works

Policy gradient methods parameterize the policy directly and optimize by gradient ascent. Actor-Critic methods use a value function to reduce variance. PPO clips updates for stability and powers RLHF in ChatGPT.

Mathematical Notation

∇J(θ) = E[∇log π_θ(a|s) · R]

The gradient of expected reward equals the expected product of the score function and the return.

Related Concepts

• Reinforcement Learning — A paradigm where an agent learns to make decisions by receiving rewards or penalties from its environment through trial and error.
• Fine-Tuning — Adapting a pre-trained model to a specific task by continuing training on a smaller, task-specific dataset.
• Q-Learning — A model-free RL algorithm that learns action values without knowing environment dynamics.