Tag
GRPO
Posts and notes tagged with GRPO.
- November 5, 202638 min read
Formal Verification for LLM Reasoning: Solvers and Theorem Provers
How solvers and proof assistants check LLM reasoning: solver feedback on partial programs, formal logic checks inside a chain of thought, and LLM theorem provers for Lean, Isabelle and Coq that retrieve, search, repair and learn from the checker.
- October 29, 202644 min read
Reward Models for Reasoning: ORMs and PRMs
How outcome and process reward models judge reasoning: scoring the final answer or every step, where their step labels come from, why GRPO already contains one, how verifiers that write and run checks work, and how to train them with few or no labels.
- October 21, 202612 min read
Contrastive Perception Policy Optimization (CPPO)
How CPPO improves perception in vision-language models during reinforcement learning: it finds the tokens whose entropy rises most when the image is hidden, and trains them with a contrastive loss to stay stable under harmless image changes and to shift when the image's content is removed.
- October 16, 202611 min read
Perception-Aware Policy Optimization (PAPO)
How PAPO teaches multimodal models to look at the image while they learn to reason: it adds a loss that rewards responses whose probability drops when the image is masked, and keeps that loss from being gamed with two entropy terms.
- October 5, 202613 min read
Visually-Perceptive Policy Optimization (VPPO)
How VPPO makes reinforcement learning for vision-language models pay attention to the image: it measures how much each token depends on the picture, scales each response's advantage by that dependency, and updates only the most image-dependent tokens.
- September 13, 202611 min read
Group Sequence Policy Optimization (GSPO)
How GSPO fixes the instability of GRPO by moving the importance ratio from tokens to whole responses: one length-normalized sequence ratio, sequence-level clipping, and equal weights for every token, which stabilizes large Mixture-of-Experts training and powered Qwen3.
- September 3, 202610 min read
Clipped IS-weight Policy Optimization (CISPO)
How CISPO, from MiniMax-M1, keeps every token in the gradient: instead of clipping token updates like PPO and GRPO, which silently drops rare reasoning tokens such as 'Wait' and 'However', it clips the importance sampling weight and lets every token keep learning.
- August 27, 202613 min read
Value-Model-Based Augmented PPO (VAPO)
How VAPO makes PPO with a value model work for long chain-of-thought reasoning: pretraining the value model, decoupling and adapting GAE to response length, and borrowing DAPO's fixes for sparse rewards, to beat value-free methods like GRPO and DAPO.
- August 16, 202612 min read
Dr. GRPO: Group Relative Policy Optimization Done Right
How Dr. GRPO fixes two biases in GRPO: dividing by the response length makes long wrong answers cheap, and dividing by the group's standard deviation overweights questions that are too easy or too hard. Removing both recovers the unbiased policy gradient and stops responses from growing needlessly long.
- July 29, 202616 min read
Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO)
How DAPO fixes the failures of GRPO in long chain-of-thought reasoning: Clip-Higher against entropy collapse, dynamic sampling against zero-gradient prompts, a token-level loss for long responses, and overlong reward shaping for truncated answers.
- June 20, 202616 min read
REINFORCE Leave-One-Out (RLOO)
How RLOO replaces PPO in RLHF with the simplest policy gradient: treating the whole response as one action, using REINFORCE, and comparing each response with the average reward of the other responses to the same prompt.
- June 10, 202618 min read
Group Relative Policy Optimization (GRPO)
How Group Relative Policy Optimization (GRPO) removes PPO's value model by comparing answers within a group, and why it works so well for training reasoning models. Part 6 of a series on policy optimization for LLMs.