Perception-Aware Policy Optimization (PAPO)

PAPO cover: the same response scored with the real image and with a masked image, pushed apart

Introduction

When reinforcement learning with verifiable rewards (RLVR) is applied to vision-language models, the training objective is usually the one built for text, such as GRPO [1]. The model writes a chain of reasoning about an image and a question, and it is rewarded only for a correct final answer. Nothing in this objective asks the model to actually look at the image.

The authors of PAPO checked where such a model goes wrong. They trained Qwen2.5-VL-3B with GRPO and sorted $200$ of its mistakes by hand into perception errors, reasoning errors, calculation errors, and inconsistencies between the reasoning and the answer [2]. Most of the mistakes, $67%$ , were perception errors: the reasoning was fine, but the model read the image wrong. In one example, it had to find a side $x$ of a right triangle with a $6 0^{\circ}$ angle and a hypotenuse of $18$ . It decided that $x$ was opposite the $6 0^{\circ}$ angle, when the figure showed it opposite the $3 0^{\circ}$ angle, and the correct answer was simply half the hypotenuse, $9$ .

In 2025, a team from the University of Illinois Urbana-Champaign and Alibaba proposed PAPO (Perception-Aware Policy Optimization) [2] to address this through the training objective itself. Its idea is to make the model learn to perceive while it learns to reason. It adds one loss to GRPO: the model should be much less confident in its response when the image is mostly hidden. A response that the model would write just as easily without the image is not grounded in it. PAPO needs no extra data, no reward model, and no stronger teacher model, and it can replace GRPO or DAPO [3] directly. It cut perception errors by $30.5%$ .

In this post, we first recall the GRPO objective, then build PAPO’s Implicit Perception Loss, see how it can be gamed and how the Double Entropy Loss prevents that, and put everything together in the PAPO objective, ending with the limitations.

The Starting Point: GRPO

A training example has an image $I$ , a question $q$ , and a short ground-truth answer $a$ . The old policy $π_{θ_{old}}$ writes a group of $G$ responses ${o_{i}}_{i = 1}^{G}$ , and a rule-based checker gives each one a reward $R_{i} = 1$ if its answer matches $a$ and $R_{i} = 0$ otherwise. As in Eq. (3) of the GRPO post, the advantage of each token of response $o_{i}$ is its reward normalized within the group:

A^_{i, t} = \frac{R _{i} - mean ( R )}{std ( R )} (1)

where $R = {R_{1}, \dots, R_{G}}$ . The policy is then trained to maximize the clipped objective with a KL penalty toward a reference model $π_{ref}$ , the same as Eq. (7) of the GRPO post:

J_{GRPO} (θ) = E [\frac{1}{G} i = 1 \sum G \frac{1}{∣ o _{i} ∣} t = 1 \sum ∣ o_{i} ∣ (min (r_{i, t} (θ) A^_{i, t}, clip (r_{i, t} (θ), 1 - ϵ_{l}, 1 + ϵ_{h}) A^_{i, t}) - β D_{KL} [π_{θ} ∥ π_{ref}])] (2)

with the probability ratio $r_{i, t} (θ) = π_{θ} (o_{i, t} ∣ q, I, o_{i, < t}) / π_{θ_{old}} (o_{i, t} ∣ q, I, o_{i, < t})$ . The paper uses the clip-higher setting from DAPO, with a larger upper bound $ϵ_{h} = 0.3$ than lower bound $ϵ_{l} = 0.2$ . The image $I$ appears only as part of the input. Whether the response actually used it makes no difference to the objective.

The Implicit Perception Loss

To reward the use of the image, PAPO needs a way to tell whether a response depends on it. It compares the probability of the same response under two inputs: the real image $I$ , and a corrupted image $I_{mask}$ in which most of the image has been hidden. Their ratio is the perception ratio [2]:

r^{prcp} (θ) = \frac{π _{θ} ( o ∣ q , I )}{π _{θ} ( o ∣ q , I _{mask} )} (3)

Both probabilities come from the same model and the same response $o$ , and only the image differs. If the response really relied on the image, the model should find it much less likely once the image is hidden, so the denominator is small and the ratio is large. If the model would have written the same response without the image, perhaps because it guessed from the question alone, the two probabilities are close and the ratio is near $1$ . So a well-behaved multimodal model should have a high perception ratio.

The logarithm of this ratio is the information gain from the image: how much more probable the response becomes when the image is visible. Averaging it over responses sampled from the model gives the KL divergence between the two predictions:

D_{KL} [π_{θ} ∥ π_{θ}^{mask}] = E_{o \sim π_{θ}} [lo g \frac{π _{θ} ( o ∣ q , I )}{π _{θ} ( o ∣ q , I _{mask} )}]

where $π_{θ}^{mask}$ is shorthand for the same model given the masked image. This is the Implicit Perception Loss. Unlike the KL penalty in Eq. (2), which is subtracted to keep the policy close to the reference model, this KL divergence is maximized: PAPO wants the predictions with and without the image to be as different as possible. It is called implicit because nothing tells the model what it should see in the image. The model is only rewarded for making its responses depend on it.

In practice, the KL divergence is computed per token from the perception ratio, with the same estimator as the KL penalty in Eq. (6) of the GRPO post, following Schulman [4]:

D_{KL} [π_{θ} ∥ π_{θ}^{mask}] \approx r_{i}^{prcp} (θ) - lo g r_{i}^{prcp} (θ) - 1 (4)

The function $x - lo g x - 1$ is never negative. Its derivative, $1 - 1/ x$ , is zero only at $x = 1$ , where the function is $0$ , and it grows on both sides. So the estimate is $0$ when the masked image changes nothing, and it grows as the perception ratio moves away from $1$ .

How the Image Is Masked

The corrupted image has to remove the information in the picture. Adding noise to the pixels is not enough: even with strong Gaussian noise, the shapes and labels in a diagram usually stay readable. PAPO therefore splits the image into patches and blacks out a fraction of them [2]. The authors tried two ways of choosing the patches: at random, or by picking the most salient patches according to the attention of DINOv2 [5], a self-supervised vision model. Random masking worked better, and it costs almost nothing. The fraction of masked patches matters too. With too little masking, the corrupted image still shows most of what matters, and with all of it masked, training became unstable. The paper uses $60%$ .

The Double Entropy Loss

Maximizing a KL divergence has a catch: the KL divergence has no upper limit. The model can make it as large as it likes by making the masked prediction terrible, rather than by making the real one better. If the model assigns almost no probability to its responses whenever the image is masked, the denominator of Eq. (3) goes to $0$ and the perception ratio explodes, whether or not the model has learned to look at the image.

The authors observed exactly this, and called it KL hacking [2]. With a large weight on the Implicit Perception Loss, training first improved and then collapsed: the reward dropped sharply, and the model started inserting tokens unrelated to the question into its reasoning. Larger models, a larger weight, and masking the whole image all made the collapse come sooner. A warning sign was that the entropy of the model’s predictions rose, both with the real image and with the masked one. The model was becoming uncertain about everything in order to inflate the KL divergence.

So PAPO adds a penalty on both entropies, the Double Entropy Loss. The entropy $H [π_{θ}]$ measures how spread out the predictions are given the real image, and $H [π_{θ}^{mask}]$ given the masked image. Each is estimated from the sampled tokens by their negative log-probability, so a confident prediction has low entropy. Subtracting both from the objective keeps the model confident with and without the image, which closes the loophole: it can no longer inflate the KL divergence by spreading its masked predictions out. Of the regularizers the authors tried, including a stronger KL penalty toward the reference model and an entropy penalty on only one of the two, the double one gave the best results.

The PAPO Objective

Adding the Implicit Perception Loss with weight $γ$ and the two entropy terms with weights $η_{1}$ and $η_{2}$ to Eq. (2) gives the full objective, written here for GRPO [2]:

J_{PAPO} (θ) = E [\frac{1}{G} i = 1 \sum G \frac{1}{∣ o _{i} ∣} t = 1 \sum ∣ o_{i} ∣ (min (r_{i, t} (θ) A^_{i, t}, clip (r_{i, t} (θ), 1 - ϵ_{l}, 1 + ϵ_{h}) A^_{i, t}) - β D_{KL} [π_{θ} ∥ π_{ref}] + γ D_{KL} [π_{θ} ∥ π_{θ}^{mask}] - η_{1} H [π_{θ}] - η_{2} H [π_{θ}^{mask}])] (5)

Each term of Eq. (5) has its own job. The clipped term learns from the reward, as in GRPO. The $β$ term keeps the policy close to the reference model. The $γ$ term, computed with Eq. (4), pushes the response to depend on the image. The two $η$ terms keep the predictions confident, so that the $γ$ term cannot be gamed. The paper uses $β = 0.01$ and $γ = 0.02$ for the $3$ -billion-parameter model, and adds the entropy terms with $η_{1} = η_{2} = 0.05$ where collapse is a risk, such as for the larger $7$ -billion-parameter model. The DAPO version of PAPO is built the same way, with DAPO’s objective in place of the clipped term and no KL penalty toward the reference model. Since that version loses the $β$ term, which also holds the policy in place, the entropy terms are essential there.

Computing the masked probabilities costs one extra forward pass over each response, with the masked image. No extra responses have to be generated, and nothing else in the training loop changes. Every response, correct or not, is still scored with the advantage of Eq. (1).

PAPO and VPPO [6] start from the same measurement, a KL divergence between predictions with the real and a masked image, but use it in opposite ways. PAPO turns it into a loss and maximizes it for every token. VPPO keeps the GRPO loss and uses the measurement to decide which responses and which tokens the reward should flow to.

Limitations of PAPO

PAPO adds an objective that can be gamed, and keeping it in check takes care. The weight $γ$ that works for a $3$ -billion-parameter model causes collapse at $7$ billion without the entropy terms, so the weights have to be tuned to the model. The Implicit Perception Loss is applied to every token, including the many that have nothing to do with the image, and the authors list a finer selection of tokens as future work. The extra forward pass makes each training step moderately slower. And the experiments cover only models of up to $7$ billion parameters from the Qwen family, so how PAPO behaves at larger scales or on other model families is not yet known.

PAPO showed that a multimodal model can be taught to look at the image without telling it what to see: it is enough to reward responses that become much less likely when the image is hidden, as long as the model is kept from making the hidden-image predictions bad on purpose.

References

Shao, Z., Wang, P., Zhu, Q., et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300.
Wang, Z., Guo, X., Stoica, S., Xu, H., Wang, H., Ha, H., Chen, X., Chen, Y., Yan, M., Huang, F., & Ji, H. (2026). Perception-Aware Policy Optimization for Multimodal Reasoning. ICLR.
Yu, Q., Zhang, Z., Zhu, R., et al. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476.
Schulman, J. (2020). Approximating KL Divergence. Blog post.
Oquab, M., Darcet, T., Moutakanni, T., et al. (2024). DINOv2: Learning Robust Visual Features without Supervision. Transactions on Machine Learning Research.
Huang, S., Qu, X., Li, Y., Luo, Y., He, Z., Liu, D., & Cheng, Y. (2026). Spotlight on Token Perception for Multimodal Reinforcement Learning. ICLR.

Navigation