Tag
Policy Optimization
Posts and notes tagged with Policy Optimization.
- August 16, 202612 min read
Dr. GRPO: Group Relative Policy Optimization Done Right
How Dr. GRPO fixes two biases in GRPO: dividing by the response length makes long wrong answers cheap, and dividing by the group's standard deviation overweights questions that are too easy or too hard. Removing both recovers the unbiased policy gradient and stops responses from growing needlessly long.
- July 29, 202616 min read
Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO)
How DAPO fixes the failures of GRPO in long chain-of-thought reasoning: Clip-Higher against entropy collapse, dynamic sampling against zero-gradient prompts, a token-level loss for long responses, and overlong reward shaping for truncated answers.
- July 21, 202614 min read
Simple Preference Optimization (SimPO)
How SimPO simplifies DPO: it replaces DPO's reward with the average log-likelihood that the model uses to generate text, which removes the reference model, and adds a target reward margin to separate preferred and rejected responses.
- June 20, 202616 min read
REINFORCE Leave-One-Out (RLOO)
How RLOO replaces PPO in RLHF with the simplest policy gradient: treating the whole response as one action, using REINFORCE, and comparing each response with the average reward of the other responses to the same prompt.
- June 10, 202618 min read
Group Relative Policy Optimization (GRPO)
How Group Relative Policy Optimization (GRPO) removes PPO's value model by comparing answers within a group, and why it works so well for training reasoning models. Part 6 of a series on policy optimization for LLMs.
- May 13, 202619 min read
Direct Preference Optimization (DPO)
How Direct Preference Optimization (DPO) aligns language models with human preferences without a reward model or reinforcement learning, by turning the RLHF objective into a simple classification loss. Part 5 of a series on policy optimization for LLMs.
- March 21, 202627 min read
Reinforcement Learning from Human Feedback (RLHF)
How Reinforcement Learning from Human Feedback (RLHF) trains language models from human preferences: collecting comparisons, training a reward model, and optimizing the model with PPO under a KL penalty. Part 4 of a series on policy optimization for LLMs.
- March 10, 202635 min read
Proximal Policy Optimization (PPO)
How Proximal Policy Optimization (PPO) achieves the stability of TRPO with a simple clipped objective and first-order optimization, explained step by step: the advantage function, the value model, Monte Carlo and temporal difference estimates, GAE, the probability ratio, and clipping. Part 3 of a series on policy optimization for LLMs.
- March 1, 202638 min read
Trust Region Policy Optimization (TRPO)
How Trust Region Policy Optimization (TRPO) keeps policy updates stable by limiting how much a model's behavior can change, explained step by step with KL divergence, importance sampling, the Fisher information matrix, and the natural gradient. Part 2 of a series on policy optimization for LLMs.
- February 14, 20268 min read
Reinforcement Learning for LLMs: SFT to Policy Gradients
Why supervised fine-tuning is not enough to align large language models, how reinforcement learning maps onto an LLM, and why the vanilla policy gradient is unstable. Part 1 of a series on policy optimization for LLMs.