Tag
DAPO
Posts and notes tagged with DAPO.
- October 16, 202611 min read
Perception-Aware Policy Optimization (PAPO)
How PAPO teaches multimodal models to look at the image while they learn to reason: it adds a loss that rewards responses whose probability drops when the image is masked, and keeps that loss from being gamed with two entropy terms.
- October 5, 202613 min read
Visually-Perceptive Policy Optimization (VPPO)
How VPPO makes reinforcement learning for vision-language models pay attention to the image: it measures how much each token depends on the picture, scales each response's advantage by that dependency, and updates only the most image-dependent tokens.
- September 3, 202610 min read
Clipped IS-weight Policy Optimization (CISPO)
How CISPO, from MiniMax-M1, keeps every token in the gradient: instead of clipping token updates like PPO and GRPO, which silently drops rare reasoning tokens such as 'Wait' and 'However', it clips the importance sampling weight and lets every token keep learning.
- August 27, 202613 min read
Value-Model-Based Augmented PPO (VAPO)
How VAPO makes PPO with a value model work for long chain-of-thought reasoning: pretraining the value model, decoupling and adapting GAE to response length, and borrowing DAPO's fixes for sparse rewards, to beat value-free methods like GRPO and DAPO.
- July 29, 202616 min read
Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO)
How DAPO fixes the failures of GRPO in long chain-of-thought reasoning: Clip-Higher against entropy collapse, dynamic sampling against zero-gradient prompts, a token-level loss for long responses, and overlong reward shaping for truncated answers.