Group Relative Policy Optimization (GRPO)

Introduction

Part 5 ended with a question. DPO showed that for preferences between two responses, we do not need reinforcement learning at all. But for tasks with an answer that can be checked, such as a math problem or a program with tests, the most useful signal is not a fixed dataset of preferred responses. It is a reward for the model’s own, newly generated attempts: write a solution, check it, and learn from the result. That brings back online reinforcement learning, and with it PPO.

PPO works, but in Part 3 we saw what it costs for language models. To compute the advantage of every token, PPO trains a value model next to the policy, and for an LLM, that value model is itself an LLM of the same size. It doubles the memory and compute of training, and it has to learn to predict the final reward of a response from every partial response, which is hard when the reward arrives only at the end.

In 2024, Group Relative Policy Optimization (GRPO) [1], introduced with the DeepSeekMath model, answered the question of whether the advantage can be computed without a value model. Its idea is simple: instead of asking a model what reward to expect for a prompt, ask the policy to answer the same prompt several times and look at how its answers actually scored. The average score of this group of answers is the expected reward, and each answer’s advantage is how much better or worse it did than its group. A year later, GRPO with rewards from simple checkers trained DeepSeek-R1 [2], one of the first open models to reason step by step at the level of the best closed models.

In this post, we derive GRPO from the PPO we already know. We first see exactly what the value model was used for, and why the group of answers can replace it. We then build the GRPO objective: PPO’s clipped objective with group advantages, and a KL penalty that is moved from the reward into the loss. Finally, we look at how GRPO is used with verifiable rewards to train reasoning models, what its limitations are, and how the whole series fits together.

What the Value Model Was For

To remove the value model, we first need to be precise about what it did. Recall from Part 3 that the advantage compares what actually happened after a token with what was expected to happen. The simplest estimate of it, the Monte Carlo advantage from Eq. (6) in Part 3, is the return minus the value model’s prediction:

$${\hat{A}}_t=G_t-V_{\varphi }(s_t)$$

For the tasks GRPO is designed for, this estimate simplifies considerably. The reward is given only once the response is finished: $r_t=0$ for every token except the last one, which gets the reward of the whole response, $r(x,y)$. With a discount factor of $\gamma =1$, the return of every token, from Eq. (3) in Part 3, is therefore the same number:

$$G_t=r_t+r_{t+1}+\cdots +r_T=0+0+\cdots +r(x,y)=r(x,y)$$

So every token sees the same outcome, the reward of the whole response. The only part of the advantage that changes from token to token is the value model’s prediction $V_{\varphi }(s_t)$, the baseline. Its job is to say what reward to expect, so that a response is pushed up only when it did better than expected, and down only when it did worse.

This is the part GRPO replaces. Training a second LLM to produce the baseline is expensive, and it is also hard: the value model has to predict the final reward of a response from every partial response, while it only ever receives a single number at the end. DeepSeekMath points out that this makes an accurate value at every token difficult to learn [1]. So the question becomes: is there another way to know what reward to expect?

The Group as the Baseline

GRPO’s answer comes from the definition of the value itself. In this section, we first write the expected reward of a prompt as an average, then estimate that average from a group of responses, normalize the result, and finally see what this means for every token.

Estimating the Expected Reward by Sampling

In Part 3, the value of a state was defined as the expected return from it. At the very start of a response, the state is just the prompt $x$, and the return is the reward of the whole response. So the value of a prompt is the average reward of the responses the policy writes to it:

$$V(x)={\mathbb{E}}_{y\sim {\pi }_{\theta }(\cdot \mid x)}[r(x,y)]\text{\hspace{1em}(1)}$$

The value model learns to predict the average in Eq. (1). But an average can also be estimated by sampling, which is the Monte Carlo idea from Part 3: let the policy write $G$ responses $y_1,\dots ,y_G$ to the same prompt, score each one, and take the mean of their rewards $r_i=r(x,y_i)$:

$$V(x)\approx \text{mean}(\mathbf{r})=\frac{1}{G}\sum _{i=1}^G{r}_i\text{\hspace{1em}(2)}$$

where $\mathbf{r}=(r_1,\dots ,r_G)$ is the vector of rewards in the group. (Following the GRPO paper, we write $G$ for the size of the group. It is not the return $G_t$ from Part 3, which we no longer need.) The more responses in the group, the closer this mean is to the true value. DeepSeekMath used groups of $G=64$ responses per prompt [1].

The idea of using other samples as the baseline is older than GRPO. It is known as a sampled baseline, and methods such as RLOO use the average reward of the other responses to the same prompt in the same way [3]. What GRPO adds is how it scales the result and how it combines it with PPO.

The Group-Relative Advantage

With the group mean of Eq. (2) as the baseline, the advantage of response $i$ is its reward minus the mean: positive if it did better than its group, negative if it did worse. GRPO then divides by the standard deviation of the rewards in the group,

$$\text{std}(\mathbf{r})=\sqrt{\frac{1}{G}\sum _{i=1}^G(r_i-\text{mean}(\mathbf{r}){)}^2}$$

which gives the group-relative advantage [1]:

$${\hat{A}}_i=\frac{r_i-\text{mean}(\mathbf{r})}{\text{std}(\mathbf{r})}\text{\hspace{1em}(3)}$$

The division puts every prompt on the same scale. For some prompts, the rewards of the group are spread far apart, and for others they are close together. Without the division, prompts with widely spread rewards would produce much larger updates than the others. After it, the advantages of every group have a standard deviation of $1$, so every prompt contributes to training on an equal footing. In practice, a small constant is added to the denominator so that it is never zero.

From Responses to Tokens

Eq. (3) gives one advantage per response, but the policy is updated token by token. GRPO assigns the advantage of a response to every one of its tokens:

$${\hat{A}}_{i,t}={\hat{A}}_i\text{for every token }t\text{ of }{y}_i\text{\hspace{1em}(4)}$$

This matches what we found for the return: when the reward arrives only at the end, every token of a response sees the same outcome. The difference from PPO is that the baseline is also shared. PPO subtracts a different $V_{\varphi }(s_t)$ at every token, while GRPO subtracts the same group mean for the whole response. This setting, with one reward for the whole response, is called outcome supervision. DeepSeekMath also describes a process supervision variant, in which a process reward model scores every reasoning step [4], the step rewards are normalized across the group in the same way, and the advantage of a token is the sum of the normalized rewards of the steps from that token on [1].

The GRPO Objective

With an advantage for every token, the rest of GRPO follows PPO closely. In this section, we first put the group advantages into PPO’s clipped objective, then add a KL penalty to keep the policy close to the reference model, and see why GRPO puts that penalty in the loss instead of in the reward.

The Clipped Objective Over a Group

Like PPO, GRPO reuses each batch of responses for several updates. So it uses the same probability ratio from Eq. (9) in Part 3 between the policy being trained and the policy that generated the group, now written for token $t$ of response $i$, where $s_{i,t}$ is the prompt and the tokens of $y_i$ before $y_{i,t}$:

$${\rho }_{i,t}(\theta )=\frac{{\pi }_{\theta }(y_{i,t}\mid s_{i,t})}{{\pi }_{{\theta }_{\text{old}}}(y_{i,t}\mid s_{i,t})}$$

and the same clipping as in Eq. (11) in Part 3, to keep each update small, with the token advantages ${\hat{A}}_{i,t}$ from Eq. (4). The only change is how the terms are averaged. The clipped term is first averaged over the $|y_i|$ tokens of each response, and then over the $G$ responses of the group:

$${\mathcal{J}}^{\text{CLIP}}(\theta )=\mathbb{E}\left[\frac{1}{G}\sum _{i=1}^G\frac{1}{|y_i|}\sum _{t=1}^{|y_i|}\min ({\rho }_{i,t}(\theta ){\hat{A}}_{i,t},\text{clip}({\rho }_{i,t}(\theta ),1-ϵ,1+ϵ\left){\hat{A}}_{i,t}\right)\right]\text{\hspace{1em}(5)}$$

where the expectation is over prompts $x$ and groups of responses sampled from ${\pi }_{{\theta }_{\text{old}}}$.

The KL Penalty, Moved into the Loss

As in RLHF, the policy should not drift too far from the reference model ${\pi }_{\text{ref}}$, so GRPO keeps a KL penalty. But it places it differently. In Part 4, the penalty was folded into the reward of every token, and from there it entered the advantages. In GRPO, that would mix the penalty into the group mean and standard deviation of Eq. (3). So GRPO keeps the advantage purely about the reward, and subtracts the KL divergence directly from the objective instead [1].

The KL divergence is estimated from the sampled tokens. The most direct estimate would be the log-ratio of the two models at each sampled token, as in RLHF. But a single log-ratio can be negative, even though a KL divergence never is, and it has a high variance. GRPO uses a different estimator [1], [5]. To write it, let $u$ be the ratio of the reference model to the policy at a sampled token:

$$u=\frac{{\pi }_{\text{ref}}(y_{i,t}\mid s_{i,t})}{{\pi }_{\theta }(y_{i,t}\mid s_{i,t})}$$

The estimator is:

$${\hat{D}}_{\text{KL}}=u-\log u-1\text{\hspace{1em}(6)}$$

It has two useful properties. First, it is unbiased: on average over tokens sampled from the policy, it equals the true KL divergence. The average of $u$ is exactly $1$, because the policy’s probabilities cancel:

$${\mathbb{E}}_{y_t\sim {\pi }_{\theta }}[u]=\sum _{y_t}{\pi }_{\theta }(y_t\mid s_t)\frac{{\pi }_{\text{ref}}(y_t\mid s_t)}{{\pi }_{\theta }(y_t\mid s_t)}=\sum _{y_t}{\pi }_{\text{ref}}(y_t\mid s_t)=1$$

so the term $u-1$ averages to $0$. The remaining term, $-\log u$, is the log-ratio of the policy to the reference model, whose average under the policy is exactly the KL divergence from Eq. (1) in Part 2:

$${\mathbb{E}}_{y_t\sim {\pi }_{\theta }}[-\log u]={\mathbb{E}}_{y_t\sim {\pi }_{\theta }}\left[\log \frac{{\pi }_{\theta }(y_t\mid s_t)}{{\pi }_{\text{ref}}(y_t\mid s_t)}\right]=D_{\text{KL}}({\pi }_{\theta }\Vert {\pi }_{\text{ref}})$$

Second, unlike a single log-ratio, it is never negative. The logarithm always lies below its tangent line at $u=1$:

$$\log u\le u-1\text{for every }u>0$$

Moving $\log u$ to the right side gives:

$$u-\log u-1\ge 0$$

with equality only when $u=1$, that is, when the two models agree on the token.

The Full Objective

Putting the clipped objective of Eq. (5) and the KL estimator of Eq. (6) together, with a coefficient $\beta$, gives the GRPO objective, which is maximized [1]:

$${\mathcal{J}}^{\text{GRPO}}(\theta )=\mathbb{E}\left[\frac{1}{G}\sum _{i=1}^G\frac{1}{|y_i|}\sum _{t=1}^{|y_i|}\left(\min \right({\rho }_{i,t}{\hat{A}}_{i,t},\text{clip}({\rho }_{i,t},1-ϵ,1+ϵ){\hat{A}}_{i,t})-\beta {\hat{D}}_{\text{KL}})\right]\text{\hspace{1em}(7)}$$

Compared with PPO, nothing in this objective needs a value model. The policy, the reference model, and a source of rewards are all it uses. Figure 1 shows the difference.

The GRPO Algorithm

The training loop has the same structure as PPO’s in Part 3. The only difference in the loop is that every prompt is answered $G$ times, and the advantages come from the group:

The savings are clear when we count the models. PPO in RLHF needs four LLMs: the policy, the value model, the reward model, and the reference model. GRPO drops the value model, leaving three. And when the reward comes from a checker instead of a reward model, only two LLMs remain, the policy and the reference model.

	PPO	GRPO
Baseline for the advantage	Value model $V_{\varphi }(s_t)$, trained alongside the policy	Mean reward of a group of responses
Advantage	Different for every token, from GAE	Same for every token of a response
Responses per prompt	One	A group of $G$
KL penalty	Folded into the token rewards	Added directly to the loss
LLMs in memory	Four, with a reward model	Three with a reward model, two with a checker

GRPO with Verifiable Rewards

GRPO works with any reward. In DeepSeekMath, it was used with a reward model, as in RLHF [1]. But it is a natural fit for rewards that need no model at all. When the answer can be checked by a program, the reward can come from a checker: compare the final answer of a math problem with the correct one, or run the tests of a program. Training with such rewards is called reinforcement learning with verifiable rewards (RLVR) [6].

Verifiable rewards remove two problems at once. They need no human labels during training, because the checker is a program. And they are much harder to hack than a reward model: as we saw in Part 4, a policy optimized against a learned reward model eventually finds its mistakes, but a correct checker has no such mistakes to find. GRPO, in turn, removes the value model, so what is left is a simple loop: generate a group of answers, check them, and push up the ones that were right more often than the group.

This combination trained DeepSeek-R1 [2]. Its first version, DeepSeek-R1-Zero, was trained with GRPO directly from a pre-trained base model, without any SFT, using only two rule-based rewards: an accuracy reward for a correct final answer, and a format reward for placing the reasoning between dedicated tags before the answer. The authors chose rule-based rewards precisely to avoid the reward hacking of neural reward models [2]. Nothing in these rewards asked for long reasoning, yet as training went on, the responses grew longer, and the model learned on its own to check its work, to reconsider earlier steps, and to try other approaches when one failed. The final DeepSeek-R1 added a small SFT stage before RL to make its reasoning more readable, and reached reasoning performance comparable to the best closed models of its time [2].

Limitations of GRPO

GRPO is simpler than PPO, but its design has costs of its own, and later work has identified several of them:

• It needs many samples. Every prompt is answered $G$ times, so generation, already the most expensive part of RL for LLMs, is multiplied by the group size.
• Uniform groups give no signal. As in the example above, when all the answers in a group are right, or all are wrong, every advantage is $0$, and the whole group is wasted. This happens more and more as the model gets better at the training problems. DAPO addresses it by filtering out such groups and sampling new prompts until every batch has useful groups [7].
• The normalizations introduce biases. Dividing by the response length $|y_i|$ in Eq. (7) makes each token of a long response count less, so a long wrong answer is penalized less per token than a short one. Dividing by the group’s standard deviation gives more weight to prompts whose answers are almost all right or almost all wrong. Dr. GRPO removes both normalizations to avoid these biases [8].
• Credit assignment is coarse. With outcome supervision, every token of a response gets the same advantage, whether it was the step that solved the problem or an irrelevant one. PPO’s value model, for all its cost, could tell them apart.

Looking Back at the Series

This post ends the series, and GRPO is a good place to look back, because it reuses almost everything that came before.

In Part 1, we saw that the policy gradient turns supervised fine-tuning into learning from rewards, by weighting each token with its advantage, and that its updates can be too large and break the model. In Part 2, TRPO fixed this with a trust region: a limit on how far the model’s outputs may move in one update, enforced with second-order optimization. In Part 3, PPO got the same stability with a simple clipped objective and first-order optimization, and we built the advantage it needs from a value model and GAE. In Part 4, RLHF gave PPO a reward for tasks that no program can check, through a reward model trained on human comparisons and a KL penalty against a reference model. In Part 5, DPO showed that for preferences, the reward model and the RL loop can be replaced by a single classification loss. And in this post, GRPO brought online RL back for tasks with checkable answers, while removing the value model that made PPO so expensive.

Each method kept what worked in the one before it, and removed what was most expensive.

References

1. Shao, Z., Wang, P., Zhu, Q., et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300.
2. DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948.
3. Ahmadian, A., Cremer, C., Gallé, M., et al. (2024). Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs. ACL.
4. Lightman, H., Kosaraju, V., Burda, Y., et al. (2023). Let’s Verify Step by Step. arXiv:2305.20050.
5. Schulman, J. (2020). Approximating KL Divergence. Blog post.
6. Lambert, N., Morrison, J., Pyatkin, V., et al. (2024). Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124.
7. Yu, Q., Zhang, Z., Zhu, R., et al. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476.
8. Liu, Z., Chen, C., Li, W., et al. (2025). Understanding R1-Zero-Like Training: A Critical Perspective. arXiv:2503.20783.