Generative Adversarial Policy Optimization (GAPO)

GAPO cover: a generator and a reward model trained against each other in an adversarial loop

Introduction

Many real uses of language models come with constraints: write a product description of at most $50$ words, mention the price, use a formal tone, do not use bullet points. A model has to read every constraint in the prompt and produce a response that satisfies all of them. Models are usually taught this in one of two ways. Supervised fine-tuning shows them examples of responses that satisfy the constraints, but it never shows them what a violation looks like, so a model can learn to imitate the style of correct responses without checking the constraints at all. Preference optimization, such as DPO [1] or PPO [2] with a reward model, shows them a better and a worse response to the same prompt, which adjusts which responses the model prefers, but it still does not directly teach it to read the constraints.

In 2025, a team from Fudan University and Alibaba proposed GAPO (Generative Adversarial Policy Optimization) [3], built on two ideas. The first is to change the preference data: instead of comparing two responses to one prompt, compare one response with two prompts, one whose constraints it satisfies and one whose constraints it violates. This forces the reward model to look at the constraints. The second is to train the reward model and the policy against each other, like the two networks of a Generative Adversarial Network (GAN) [4]: the reward model learns to catch the policy’s mistakes, and the policy learns to avoid them. On instruction-following benchmarks, this beat PPO, DPO, and other preference methods.

In this post, we go through these two ideas and the adversarial loop that ties them together, then look at the results and the limitations. The reward model and PPO that GAPO builds on are covered in the RLHF post.

Preferential Prompts Instead of Preferential Responses

Let us first write the problem down. A prompt $P = (T, C)$ has two parts: a free-text task $T$ , such as “write a description of this phone”, and a set of constraints $C = {C_{1}, \dots, C_{n}}$ , such as “at most $50$ words” and “mention the price”. A response $R$ either satisfies a constraint or it does not, which GAPO captures with a constraint satisfaction function [3]:

L (R, C_{i}) = {10 if R ⊨ C_{i} otherwise (1)

The language model, which GAPO calls the generator $π_{θ}$ , should write responses that satisfy as many constraints as possible. Counting the satisfied constraints with Eq. (1) and averaging over the responses the generator writes gives the objective:

E (π_{θ}) = E_{R \sim π_{θ} (P)} [C_{i} \in C \sum L (R, C_{i})] (2)

The question is what data teaches a model this. The usual preference data, as in the RLHF post, pairs one prompt with a preferred and a rejected response. GAPO reverses this. It keeps the response fixed and changes the prompt. It starts from a dataset $D = {(P_{i}, R_{i})}_{i = 1}^{N}$ in which each response satisfies every constraint of its prompt, and builds a rejected version of each prompt, one that the same response violates. There are two ways to do this.

Constraint modification takes one constraint $C_{i, j}$ and changes it until the response no longer satisfies it. A response of $40$ words satisfies “at most $50$ words”, but not “at most $20$ words”:

C_{i, j}^{reject} = f_{modify} (C_{i, j}), where L (R_{i}, C_{i, j}^{reject}) = 0 (3)

Constraint insertion keeps every constraint and adds a new one that the response violates. A response that mentions the price violates “do not mention the price”:

C_{i}^{reject} = C_{i} \cup {C_{i, n + 1}^{reject}}, where L (R_{i}, C_{i, n + 1}^{reject}) = 0 (4)

Either way, Eq. (3) or Eq. (4), the rejected constraint set contains at least one constraint that the response breaks:

\exists C_{i, j}^{reject} \in C_{i}^{reject} : L (R_{i}, C_{i, j}^{reject}) = 0 (5)

So every example now comes in two versions with the same response: an accepted prompt $P_{i}^{accept} = (T_{i}, C_{i})$ , which the response satisfies, and a rejected prompt $P_{i}^{reject} = (T_{i}, C_{i}^{reject})$ , which it violates. Together they form the augmented dataset:

D^{'} = {(P_{i}^{accept}, R_{i}), (P_{i}^{reject}, R_{i})}_{i = 1}^{N} (6)

Because the response is the same in both pairs, its style or its length says nothing about which pair is which. Eq. (5) tells us the difference lies in the constraints, so the only way to tell the pairs apart is to read the constraints and check them against the response. A reward model trained on Eq. (6) has to learn what each constraint means, which is exactly what the objective in Eq. (2) asks for.

An Encoder-Only Reward Model

Checking a response against its constraints means comparing words in the prompt with words in the response. This is where the architecture of the reward model matters. A decoder-only language model, the usual choice for a reward model, reads the text from left to right: each token can only attend to the tokens before it. When it reads a constraint in the prompt, the response has not appeared yet, so the constraint can only be checked later, from the response tokens looking back.

GAPO uses an encoder-only model instead, in which every token attends to every other token in both directions. Each constraint can be compared directly with every part of the response, and each part of the response with every constraint. The authors used Longformer [5], an encoder for long texts, with about $0.4$ billion parameters, much smaller than the $7$ -billion-parameter policy it trains [3].

The paper writes the reward model as $R (c, t)$ , not to be confused with a response $R$ . It is a classifier: given a prompt $c$ and a response $t$ , it outputs the probability that the response satisfies the prompt’s constraints. Each training example carries a label $y$ , which is $1$ for a match and $0$ for a mismatch, and the reward model is trained with the binary cross-entropy loss [3]:

L_{R} (θ) = - E_{(c, t, y) \sim D^{'}} [y lo g R (c, t) + (1 - y) lo g (1 - R (c, t))] (7)

The Adversarial Training Loop

With the data and the reward model in place, GAPO trains the reward model and the generator in turns, like the discriminator and the generator of a GAN. In a GAN, the discriminator learns to tell real examples from the generator’s fakes, and the generator learns to produce examples the discriminator cannot tell apart from real ones. In GAPO, the reward model plays the discriminator, and the language model is the generator.

Training begins with a warmup. The reward model is trained alone with Eq. (7) on the accepted and rejected pairs of Eq. (6), so that it has learned to check constraints before it starts judging the generator. Then the adversarial phase alternates between a reward model step and a generator step.

The Reward Model Step

The generator writes a response $R^_{i}$ to each prompt $P_{i}$ . These responses join the reward model’s training data as negatives, labeled $0$ , next to the accepted pairs labeled $1$ and the rejected prompts labeled $0$ [3]:

D^{'} = {(P_{i}^{accept}, R_{i}, 1), (P_{i}^{reject}, R_{i}, 0), (P_{i}, R^_{i}, 0)} (8)

The reward model is then trained on this data with Eq. (7). Labeling the generator’s own responses as negatives is what makes the loop adversarial. The reward model is always trained to rank the generator’s current responses below the reference responses, so it keeps learning to spot exactly the mistakes the generator is making now.

The Generator Step

Now the reward model is held fixed, and the generator learns to score higher under it. The generator writes a response one token at a time: $π_{θ} (t_{n} ∣ c_{n})$ is the probability of the token $t_{n}$ at position $n$ , given the context $c_{n}$ , which is the prompt plus the tokens written so far. GAPO updates it with PPO [2], using the objective [3]:

L_{G} (θ) = E_{n} [\frac{π _{θ} ( t _{n} ∣ c _{n} )}{π _{ref} ( t _{n} ∣ c _{n} )} A_{n}] (9)

Each token’s probability ratio against the reference generator $π_{ref}$ is weighted by its advantage $A_{n}$ . As in the PPO post, the advantage says how much better a token turned out than what was expected from its context:

A_{n} = Q^{π} (c_{n}, t_{n}) - V^{π} (c_{n}) (10)

Here $V^{π} (c)$ is the value function, the reward expected from context $c$ onward, and $Q^{π} (c, t)$ is the action value, the reward expected after writing token $t$ in context $c$ . The action value is the reward model’s score for the token plus the discounted value of the context it leads to:

Q^{π} (c_{n}, t_{n}) = R (c_{n}, t_{n}) + γ E_{c_{n + 1} \sim π_{θ}} [V^{π} (c_{n + 1})] (11)

Putting Eq. (11) into Eq. (10) gives:

A_{n} = R (c_{n}, t_{n}) + γ E_{c_{n + 1} \sim π_{θ}} [V^{π} (c_{n + 1})] - V^{π} (c_{n})

This is the TD error from Eq. (7) of the PPO post, with the reward model’s score playing the role of the reward. The value function itself is learned alongside the generator, by minimizing its squared error against the reward:

L_{V} (θ) = E_{c} [(V^{π} (c) - R (c, t))^{2}] (12)

So a generator step updates $π_{θ}$ with Eq. (9) and $V^{π}$ with Eq. (12), and the loop goes back to the reward model.

Why the Loop Works

As the generator improves, its responses become harder to tell apart from the reference responses, and the reward model, trained on them as negatives through Eq. (8), has to learn finer distinctions. The difficulty of the training data increases on its own, without anyone writing harder examples by hand [3]. This also solves a practical problem of RLHF. Normally, the reward model has to be trained well before PPO begins, and a weak reward model is exploited by the policy, as we saw with reward hacking. In GAPO, the reward model keeps training on the generator’s latest responses, so any weakness the generator starts exploiting becomes new training data for the reward model.

The full procedure follows the paper’s algorithm [3]:

Algorithm 1 Generative Adversarial Policy Optimization

1 : 2 : 3 : 4 : 5 : 6 : 7 : 8 : 9 : 10 : 11 : build D^{'} with rejected prompts, Eq. (3) to Eq. (6) for t = 1, \dots, T_{warmup} do (warmup) sample a batch (P_{i}, R_{i}) from D update R (c, t) with Eq. (7) on (P_{i}^{accept}, R_{i}, 1), (P_{i}^{reject}, R_{i}, 0) for t = T_{warmup} + 1, \dots, T_{warmup} + T do (adversarial) if t mod 2 = 1 then sample a batch (P_{i}, R_{i}) from D and generate R^_{i} = π_{θ} (P_{i}) update R (c, t) with Eq. (7) on the data of Eq. (8) else update π_{θ} with Eq. (9), using A_{n} from Eq. (10) and Eq. (11) update V^{π} with Eq. (12)

Results

The authors trained Qwen2.5-7B [6] on two tasks: the IFEval instruction-following benchmark [7], and a product-description dataset they built, where descriptions must follow constraints on product attributes [3]. On IFEval, the overall accuracy was [3]:

Method	IFEval (all)
Base model	57.8
SFT	78.3
PPO	75.6
KTO [8]	54.4
DPO	33.3
SimPO [9]	30.6
GAPO	83.9

On the product-description task, GAPO scored $95.4$ with an automatic reward-model judge, against $89.4$ for PPO, and $89$ against $81$ in a human evaluation. The offline preference methods, DPO, SimPO, and ORPO, did poorly on both tasks, below even the base model. A likely reason is that they are built to compare two responses to the same prompt, and the preferential-prompt data, where the two examples share one response and differ in the prompt, does not fit that form. PPO and GAPO, which learn from a reward model, can use it.

Limitations of GAPO

GAPO’s results come from a single policy, Qwen2.5-7B, on constraint-following tasks, and one of its two evaluation datasets was built by the authors themselves, so how well it carries over to other models and tasks is not yet known. The IFEval test split it reports is small. The adversarial loop also inherits the difficulties of GANs: the two models have to stay balanced, since a reward model that becomes too strong or too weak relative to the policy can stall training. And because every response the policy writes is labeled as a negative, even a response that fully satisfies its constraints is treated as worse than the reference, which ties the policy closely to the style of the reference responses.

GAPO showed that to teach a model to follow constraints, the preference data should differ in the constraints themselves, and that a reward model kept learning against the policy can supply ever harder examples on its own.

References

Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347.
Gu, Z., Chen, X., Shi, X., Wang, T., Zheng, S., Li, T., Feng, H., & Xiao, Y. (2025). GAPO: Learning Preferential Prompt through Generative Adversarial Policy Optimization. arXiv:2503.20194.
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., et al. (2014). Generative Adversarial Networks. NeurIPS.
Beltagy, I., Peters, M. E., & Cohan, A. (2020). Longformer: The Long-Document Transformer. arXiv:2004.05150.
Qwen Team (2024). Qwen2.5 Technical Report. arXiv:2412.15115.
Zhou, J., Lu, T., et al. (2023). Instruction-Following Evaluation for Large Language Models. arXiv:2311.07911.
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., & Kiela, D. (2024). KTO: Model Alignment as Prospect Theoretic Optimization. ICML.
Meng, Y., Xia, M., & Chen, D. (2024). SimPO: Simple Preference Optimization with a Reference-Free Reward. NeurIPS.

Navigation