Clipped IS-weight Policy Optimization (CISPO)

Introduction

When MiniMax trained MiniMax-M1 [1], a large reasoning model built on a Mixture-of-Experts architecture with a hybrid attention mechanism, they started with GRPO [2], the standard algorithm for training reasoning models with reinforcement learning. It did not work well. Training was slow, and the model did not develop the long chains of reasoning, with self-checking and reconsidering, that RL is supposed to bring out. After a series of ablations, they found the cause in the part of the algorithm that is supposed to keep training stable: clipping.

The tokens that start a new line of reasoning, words like “However”, “Wait”, or “Recheck”, are rare, so the model gives them low probabilities. When an update raises the probability of such a token, its ratio to the old policy quickly becomes large, the clipping of PPO and GRPO kicks in, and from then on the token contributes nothing to the gradient. The tokens that matter most for reasoning were the ones being silently dropped. MiniMax’s fix, CISPO (Clipped IS-weight Policy Optimization), clips something else: instead of clipping the update of each token, it clips the importance sampling weight, so every token keeps contributing to the gradient, with a capped weight. With it, training on the same data matched DAPO’s performance in half the training steps, and the full RL training of MiniMax-M1 took three weeks on $512$ GPUs.

In this post, we look at why PPO-style clipping drops tokens, derive the REINFORCE objective that CISPO starts from, and see exactly how clipping the weight differs from clipping the update. We then cover the unified formulation, the results, and the limitations. The PPO post covers the clipped objective that CISPO changes.

Why Clipping Drops Tokens

PPO and GRPO reuse each batch of responses for several gradient updates. To correct for the policy changing during those updates, each token’s term is multiplied by its importance sampling (IS) weight, the ratio of the current policy to the policy that generated the response:

$$r_{i,t}(\theta )=\frac{{\pi }_{\theta }(y_{i,t}\mid x,y_{i,<t})}{{\pi }_{{\theta }_{\text{old}}}(y_{i,t}\mid x,y_{i,<t})}\text{\hspace{1em}(1)}$$

and the objective takes the minimum of the weighted term and its clipped version, as in Eq. (11) of the PPO post:

$$\min (r_{i,t}(\theta ){\hat{A}}_{i,t},\text{clip}(r_{i,t}(\theta ),1-ϵ,1+ϵ\left){\hat{A}}_{i,t}\right)$$

As we derived in the PPO post, when a token has a positive advantage and its weight has risen above $1+ϵ$, the minimum picks the clipped term, $(1+ϵ){\hat{A}}_{i,t}$. That term is a constant, so its gradient is zero: the token is removed from the update. This is intended. It stops the policy from pushing a token’s probability up too far on old data.

The problem is which tokens this affects. The weight in Eq. (1) is a ratio, so how quickly it reaches $1+ϵ$ depends on how small the old probability was. With the usual $ϵ=0.2$, take a rare token whose probability rises from $0.02$ to $0.025$ after one update, and a common token whose probability rises from $0.6$ to $0.65$:

$$\frac{0.025}{0.02}=1.25>1.2,\frac{0.65}{0.6}\approx 1.08<1.2$$

The common token moved ten times more in probability, from $0.6$ to $0.65$, and is still inside the clipping range. The rare token moved by only $0.005$, and its ratio is already past $1.2$, so it is clipped and its gradient becomes zero. MiniMax found that the tokens behind reflective reasoning, such as “However”, “Recheck”, “Wait”, and “Aha”, were exactly such tokens [1]. They are rare, and the base model gives them low probabilities, but they act as forks in the reasoning: the points where the model stops to check its work or tries another approach. After the first update on a batch raised their probabilities, their weights exceeded $1+ϵ$, and for every later update on the same batch they were clipped out. MiniMax used $16$ rounds of updates per batch, so these tokens took part in one update out of sixteen. Low-probability tokens like these are also known to be important for keeping the policy’s entropy stable and for RL to keep improving as it is scaled up [3], [4].

DAPO’s Clip-Higher [5], covered in the DAPO post, addresses the same problem by raising the upper bound. MiniMax found it less effective in their setting [1]: with $16$ rounds of updates per batch, the weights of rare tokens still pass any reasonable bound, and once they do, the tokens are dropped just the same.

From REINFORCE to CISPO

CISPO starts from a different form of the objective, in which clipping can be applied without removing tokens. In this section, we write the off-policy REINFORCE objective, see that its gradient is the same as PPO’s before clipping, and then apply the clip to the weight.

The Off-Policy REINFORCE Objective

The REINFORCE gradient, derived in the RLOO post, pushes up the log-probability of each token in proportion to its advantage. When the responses come from the old policy, each token is reweighted by its IS weight. This can be written as an objective whose gradient is exactly that [1]:

$${\mathcal{J}}_{\text{REINFORCE}}(\theta )=\mathbb{E}\left[\frac{1}{|y_i|}\sum _{t=1}^{|y_i|}\text{sg}(r_{i,t}(\theta )){\hat{A}}_{i,t}\log {\pi }_{\theta }(y_{i,t}\mid x,y_{i,<t})\right]\text{\hspace{1em}(2)}$$

where $\text{sg}[\cdot ]$, stop-gradient, uses the value of the weight but does not differentiate through it. The gradient therefore acts only on the log-probability. Writing ${\pi }_{\theta }$ and ${\pi }_{{\theta }_{\text{old}}}$ for the probabilities of token $y_{i,t}$, the gradient of one term is:

$${\nabla }_{\theta }(\text{sg}(r_{i,t}){\hat{A}}_{i,t}\log {\pi }_{\theta })=r_{i,t}{\hat{A}}_{i,t}{\nabla }_{\theta }\log {\pi }_{\theta }$$

Using ${\nabla }_{\theta }\log {\pi }_{\theta }={\nabla }_{\theta }{\pi }_{\theta }/{\pi }_{\theta }$ and the definition of $r_{i,t}$, the ${\pi }_{\theta }$ cancels:

$$r_{i,t}{\hat{A}}_{i,t}{\nabla }_{\theta }\log {\pi }_{\theta }=\frac{{\pi }_{\theta }}{{\pi }_{{\theta }_{\text{old}}}}{\hat{A}}_{i,t}\frac{{\nabla }_{\theta }{\pi }_{\theta }}{{\pi }_{\theta }}={\hat{A}}_{i,t}\frac{{\nabla }_{\theta }{\pi }_{\theta }}{{\pi }_{{\theta }_{\text{old}}}}={\nabla }_{\theta }\left(r_{i,t}{\hat{A}}_{i,t}\right)$$

So before any clipping, the objective in Eq. (2) has exactly the same gradient as the ratio-weighted term $r_{i,t}{\hat{A}}_{i,t}$ of PPO and GRPO. The two objectives differ only in where the weight sits: in PPO, the ratio is part of what is differentiated; here, it is a fixed number multiplying the gradient of the log-probability.

Clipping the Weight

This form makes a different kind of clipping possible. Instead of clipping the update, CISPO clips the value of the IS weight, before it multiplies the gradient [1]:

$${\hat{r}}_{i,t}(\theta )=\text{clip}(r_{i,t}(\theta ),1-{ϵ}_{\text{low}}^{\text{IS}},1+{ϵ}_{\text{high}}^{\text{IS}})\text{\hspace{1em}(3)}$$

Using the clipped weight of Eq. (3), and adopting GRPO’s group-relative advantage and DAPO’s token-level averaging, the CISPO objective is [1]:

$${\mathcal{J}}_{\text{CISPO}}(\theta )=\mathbb{E}\left[\frac{1}{{\sum }_{i=1}^G|y_i|}\sum _{i=1}^G\sum _{t=1}^{|y_i|}\text{sg}({\hat{r}}_{i,t}(\theta )){\hat{A}}_{i,t}\log {\pi }_{\theta }(y_{i,t}\mid x,y_{i,<t})\right]\text{\hspace{1em}(4)}$$

The difference from PPO is in what happens to a token whose weight has grown too large. Take a token with a positive advantage whose weight has passed $1+ϵ$. Its gradient in each method is:

$$\begin{array}{rl}\text{PPO / GRPO:}& 0\\ \text{CISPO:}& (1+{ϵ}_{\text{high}}^{\text{IS}}){\hat{A}}_{i,t}{\nabla }_{\theta }\log {\pi }_{\theta }(y_{i,t}\mid x,y_{i,<t})\end{array}$$

PPO removes the token from the update. CISPO keeps it, and only limits how much weight it gets. A rare reasoning token whose probability is rising therefore keeps learning in all $16$ rounds of updates, but its influence on each update is capped, so it cannot dominate the gradient. MiniMax found that this also keeps the entropy of the policy in a healthy range, without the collapse that clipping can cause [1]. In their experiments, they effectively removed the lower bound by setting ${ϵ}_{\text{low}}^{\text{IS}}$ to a large value, and tuned only the upper bound.

Without the clip, Eq. (4) is the ordinary policy gradient. With it, the gradient is slightly biased, because the weights of some tokens are capped below their true values. MiniMax accepted this bias in exchange for keeping every token and reducing the variance that very large weights would cause. Like DAPO, CISPO also uses dynamic sampling and a length penalty, and it has no KL penalty [1], [5].

A Unified Formulation

The two kinds of clipping can be written in one objective by adding a mask $M_{i,t}$ to Eq. (4), which decides whether each token’s gradient is kept [1]:

$${\mathcal{J}}_{\text{unify}}(\theta )=\mathbb{E}\left[\frac{1}{{\sum }_{i=1}^G|y_i|}\sum _{i=1}^G\sum _{t=1}^{|y_i|}\text{sg}({\hat{r}}_{i,t}(\theta )){\hat{A}}_{i,t}\log {\pi }_{\theta }(y_{i,t}\mid x,y_{i,<t})M_{i,t}\right]\text{\hspace{1em}(5)}$$

Setting the mask to the condition under which PPO’s minimum picks the clipped term recovers PPO’s behavior:

$$M_{i,t}=\{\begin{array}{ll}0& \text{if }{\hat{A}}_{i,t}>0\text{ and }{r}_{i,t}(\theta )>1+{ϵ}_{\text{high}}\\ 0& \text{if }{\hat{A}}_{i,t}<0\text{ and }{r}_{i,t}(\theta )<1-{ϵ}_{\text{low}}\\ 1& \text{otherwise}\end{array}$$

CISPO is the case $M_{i,t}=1$ for every token: no token is ever dropped. Eq. (5) makes clear that “clipping” in PPO and GRPO really does two things at once, limiting the weight and dropping the token, and that CISPO keeps the first and removes the second.

Results

MiniMax compared CISPO with GRPO and DAPO by training Qwen2.5-32B from its base model on DAPO’s math dataset, and measuring accuracy on AIME 2024 [1]. At the same number of training steps, CISPO outperformed both. It also reached DAPO’s level of performance with only half of DAPO’s training steps.

CISPO was then used to train MiniMax-M1 itself, a model with $456$ billion parameters, $45.9$ billion of them active for each token. Together with the efficiency of its hybrid attention for long generations, CISPO let the full RL training finish in three weeks on $512$ H800 GPUs, at a rental cost of about 534,700 US dollars. The resulting models were comparable to or better than strong open-weight models such as the original DeepSeek-R1 [6] and Qwen3-235B [7] on standard benchmarks, with particular strengths in long-context tasks and software engineering [1].

Limitations of CISPO

CISPO gives up something that PPO’s clipping provides: a trust region. Because no token is ever dropped, a token whose weight has passed the bound keeps being pushed in every round of updates, with a capped but non-zero weight, so on a batch reused many times its probability can keep moving further from the old policy than PPO would allow. The capped weights also make the gradient biased, as the authors note. The upper bound ${ϵ}_{\text{high}}^{\text{IS}}$ becomes the main knob controlling stability and has to be tuned for each setting. And the evidence comes from MiniMax’s own experiments: the comparison with GRPO and DAPO is a single training run on one model and dataset, reported as a training curve, in a paper whose main subject is the MiniMax-M1 model rather than the algorithm.

CISPO showed that PPO’s clipping does two things at once, limiting a token’s weight and removing it from the gradient, and that in long reasoning the second is harmful: the rare tokens it removes are the ones that make reasoning possible.

References

1. MiniMax (2025). MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention. arXiv:2506.13585.
2. Shao, Z., Wang, P., Zhu, Q., et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300.
3. Cui, G., Zhang, Y., et al. (2025). The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models. arXiv:2505.22617.
4. Wang, S., Yu, L., et al. (2025). Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning. arXiv:2506.01939.
5. Yu, Q., Zhang, Z., Zhu, R., et al. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476.
6. DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948.
7. Qwen Team (2025). Qwen3 Technical Report. arXiv:2505.09388.