Tag
Long Chain-of-Thought
Posts and notes tagged with Long Chain-of-Thought.
- August 27, 202613 min read
Value-Model-Based Augmented PPO (VAPO)
How VAPO makes PPO with a value model work for long chain-of-thought reasoning: pretraining the value model, decoupling and adapting GAE to response length, and borrowing DAPO's fixes for sparse rewards, to beat value-free methods like GRPO and DAPO.
- July 29, 202616 min read
Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO)
How DAPO fixes the failures of GRPO in long chain-of-thought reasoning: Clip-Higher against entropy collapse, dynamic sampling against zero-gradient prompts, a token-level loss for long responses, and overlong reward shaping for truncated answers.