Intermediate · Reinforcement Learning
Reinforcement Learning
Visual diagram · Math · Worked example · 3 difficulty levels.
TL;DR. Reinforcement learning trains AI agents to make decisions by rewarding good actions and penalizing bad ones through trial and error in an environment.
Intuitive Explanation
Teaching a dog tricks: you don't explain the rules — the dog tries things, and when it does the right action, it gets a treat (reward). After many treats and no-treats, the dog learns which behaviors lead to rewards. RL agents learn the same way, through millions of trials.
Technical Definition
Reinforcement learning is a computational framework where an agent learns a policy π: S → A by interacting with an environment modeled as a Markov Decision Process (MDP). The agent maximizes the expected cumulative discounted reward: E[Σ_t γᵗr_t], where γ ∈ [0,1) is the discount factor.
How it works
RL is the paradigm behind game-playing AI, robotics, and LLM alignment. **Core Concepts:** - **State (s):** Current situation the agent observes - **Action (a):** Decision the agent makes - **Reward (r):** Scalar feedback signal - **Policy (π):** Strategy mapping states to actions - **Value Function V(s):** Expected future reward from state s - **Q-Function Q(s,a):** Expected future reward from taking action a in state s **Algorithm Families:** 1. **Value-Based (DQN):** Learn Q(s,a), act greedily. Works for discrete actions. 2. **Policy Gradient (REINFORCE, PPO):** Directly optimize the policy. Works for continuous actions. 3. **Actor-Critic (A3C, SAC):** Combine both — actor learns policy, critic evaluates it. 4. **Model-Based (MuZero):** Learn a world model, plan ahead. **Key Challenges:** - **Exploration vs Exploitation:** Try new things vs use what works? - **Credit Assignment:** Which action 100 steps ago caused today's reward? - **Sample Efficiency:** Real-world interactions are expensive - **Reward Design:** Wrong reward functions lead to reward hacking **Landmark Achievements:** - AlphaGo (2016): Beat world champion at Go - OpenAI Five (2019): Beat world champions at Dota 2 - RLHF (2022+): Aligns LLMs with human preferences (ChatGPT)
Mathematical Notation
Q(s, a) ← Q(s, a) + α[r + γ max_a' Q(s', a') − Q(s, a)]The Q-learning update rule: α is the learning rate, r is the immediate reward, γ is the discount factor (how much future rewards matter), and max_a' Q(s', a') is the best estimated future value from the next state.
Visual Explanation (cycle)
Agent → Action → Environment → State + Reward → Agent (loop)
Real-World Use Cases
- DeepMind: AlphaGo/AlphaZero mastered Go, Chess, and Shogi through self-play RL
- OpenAI: RLHF aligns ChatGPT with human preferences through reward modeling
- Google: RL optimizes data center cooling, reducing energy by 30%
- Waymo: RL helps train driving policies in simulation before real-world deployment
Related Concepts
- Neural Network — A computing system inspired by biological neural networks that learns patterns from data through interconnected layers of nodes.
- Fine-Tuning — Adapting a pre-trained model to a specific task by continuing training on a smaller, task-specific dataset.