Tag
PPO
Posts and notes tagged with PPO.
- August 16, 202612 min read
Dr. GRPO: Group Relative Policy Optimization Done Right
How Dr. GRPO fixes two biases in GRPO: dividing by the response length makes long wrong answers cheap, and dividing by the group's standard deviation overweights questions that are too easy or too hard. Removing both recovers the unbiased policy gradient and stops responses from growing needlessly long.
- July 29, 202616 min read
Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO)
How DAPO fixes the failures of GRPO in long chain-of-thought reasoning: Clip-Higher against entropy collapse, dynamic sampling against zero-gradient prompts, a token-level loss for long responses, and overlong reward shaping for truncated answers.
- June 20, 202616 min read
REINFORCE Leave-One-Out (RLOO)
How RLOO replaces PPO in RLHF with the simplest policy gradient: treating the whole response as one action, using REINFORCE, and comparing each response with the average reward of the other responses to the same prompt.
- June 10, 202618 min read
Group Relative Policy Optimization (GRPO)
How Group Relative Policy Optimization (GRPO) removes PPO's value model by comparing answers within a group, and why it works so well for training reasoning models. Part 6 of a series on policy optimization for LLMs.
- March 21, 202627 min read
Reinforcement Learning from Human Feedback (RLHF)
How Reinforcement Learning from Human Feedback (RLHF) trains language models from human preferences: collecting comparisons, training a reward model, and optimizing the model with PPO under a KL penalty. Part 4 of a series on policy optimization for LLMs.
- March 10, 202635 min read
Proximal Policy Optimization (PPO)
How Proximal Policy Optimization (PPO) achieves the stability of TRPO with a simple clipped objective and first-order optimization, explained step by step: the advantage function, the value model, Monte Carlo and temporal difference estimates, GAE, the probability ratio, and clipping. Part 3 of a series on policy optimization for LLMs.