Reinforcement Learning for LLMs: SFT to Policy Gradients

Introduction

Training a large language model (LLM) isn’t just about feeding it the entire internet and letting it learn. Pre-training gives the model its raw knowledge, but a model that has learned to predict the next token on Wikipedia, GitHub, and Reddit still doesn’t know how to talk to humans. It has to be guided to understand what we prefer: how we think, act, and communicate. That’s what post-training does, and it’s how LLMs went from simple next-token predictors to the controllable, safe, and reasoning-capable assistants we use today [1].

The story of how we got here starts with reinforcement learning. Early policy gradient methods were unstable: a single update that was too large could break the policy and make training collapse. Trust Region Policy Optimization (TRPO) [2] addressed this by keeping every update inside a “trust region,” but it required Fisher Information Matrices and second-order optimization, which were hard to implement and expensive to run.

Proximal Policy Optimization (PPO) [3] came next with a simpler idea: instead of solving for a trust region, clip the update. This made stable training cheap enough to scale, and PPO became the standard algorithm behind the first chat assistants. But it still needed a reward model, a value model, and a lot of GPUs. Direct Preference Optimization (DPO) [4] removed reinforcement learning entirely and learned directly from human preference pairs. Then Group Relative Policy Optimization (GRPO) [5] brought RL back in a lighter form: it removed the value model and compared answers within a group, which worked well for training reasoning models.

Each algorithm solved a problem the previous one left behind.

This series goes through each algorithm in order: why it was created and which mathematical bottleneck it solved. Each part builds on the previous one:

1. Part 1 (this post): how SFT works, how RL maps onto an LLM, and why the simplest RL method, the policy gradient, is unstable.
2. Part 2: TRPO, the first fix for that instability.
3. Part 3: PPO, which makes TRPO’s idea simple and scalable.
4. Part 4: RLHF, the full pipeline that trains LLMs with PPO from human preferences.
5. Part 5: DPO, which removes reinforcement learning from RLHF altogether.
6. Part 6: GRPO, which brings RL back without the value model.

From Pre-training to SFT

After pre-training, the first step is usually Supervised Fine-Tuning (SFT). We collect a dataset of prompts $x$ paired with good, human-written answers $y$, and train the model to imitate them. It uses the same next-token prediction as pre-training, but on curated examples instead of raw internet data:

In other words, SFT makes every token in the reference answer more likely. With enough examples, the model learns to follow instructions and answer in the expected format [1].

But SFT can only copy. It treats every token in the reference answer as equally good and never sees bad answers, so it cannot learn why one answer is better than another. Writing high-quality answers for every type of prompt is also slow and expensive.

Instead of showing the model the correct answer, we can let it generate its own and tell it how good it was.

This is what reinforcement learning does.

RL in LLM Terms

The table below maps standard RL notation to a language model. We use it throughout the post.

The goal is to make the model produce better answers (higher reward) more often, without drifting too far from ${\pi }_{\text{ref}}$. This constraint matters because the reward model is itself a neural network and can be exploited. Without it, the model may learn outputs that get high scores without being good answers, like a student who learns to pass the test without learning the subject. This is called reward hacking [6].

The simplest method for this is the vanilla policy gradient [7]. If an answer scores better than expected (positive advantage), the probability of its tokens is increased. If it scores worse than expected, the probability is decreased. Repeating this over many samples gradually improves the model.

Formally:

Each term has a simple meaning:

• ${\nabla }_{\theta }\log {\pi }_{\theta }(y_t\mid x,y_{<t})$ is the direction that makes token $y_t$ more likely.
• $A_t$ says how much and which way to move: a positive advantage pushes the token up, a negative one pushes it down.
• ${\sum }_{t=1}^T$ does this for every token in the response, and $\mathbb{E}[\cdot ]$ averages over many prompts and responses.

How is the advantage $A_t$ calculated? It compares what actually happened with what the model expected to happen. The expected part comes from the value model $V(s_t)$, a separate network trained to predict the final reward of a response from its current state $s_t$ (the prompt plus the tokens so far). In the simplest version, the advantage of token $y_t$ is the reward the response actually received minus this prediction:

Without the value model, every token in a response with a positive reward would be pushed up, even if the response was worse than usual. The value model acts as a baseline that makes the signal fairer. In practice, PPO uses a refined version of Eq. (3) called Generalized Advantage Estimation (GAE), which we will see later.

Compared with the SFT loss in Eq. (1), the policy gradient is SFT with each token weighted by its advantage $A_t$. SFT increases the probability of every token, while the policy gradient increases it for good tokens and decreases it for bad ones. The answers also come from the model itself rather than from humans.

The parameters are then updated in this direction, $\theta \leftarrow \theta +\alpha {\nabla }_{\theta }J(\theta )$, where $\alpha$ is the learning rate.

However, the gradient only gives the direction of the update, not how far to go. If the step is too small, learning is slow. If it is too large, the policy can break. In supervised learning, the next batch of ground-truth data corrects a bad update. In RL, the policy generates its own training data, so a bad update produces bad answers, which become the next training batch. The model can then fall into a death spiral from which it does not recover.

One step too large, and the policy can collapse.

This is the problem TRPO was built to solve, and it is the topic of Part 2.

References

1. Ouyang, L., Wu, J., Jiang, X., et al. (2022). Training language models to follow instructions with human feedback. NeurIPS.
2. Schulman, J., Levine, S., Moritz, P., Jordan, M. I., & Abbeel, P. (2015). Trust Region Policy Optimization. ICML.
3. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347.
4. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS.
5. Shao, Z., Wang, P., Zhu, Q., et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300.
6. Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., & Irving, G. (2019). Fine-Tuning Language Models from Human Preferences. arXiv:1909.08593.
7. Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8, 229–256.