Reinforcement Learning from Human Feedback (RLHF)

Introduction

Part 3 ended with a question. PPO can train a language model with any reward we give it, and it does so stably. But it says nothing about where that reward comes from. For a math problem, the answer is easy: a checker compares the model’s final answer with the correct one and gives $+1$ or $0$. But most of what we want from a chat assistant cannot be checked this way. Is this explanation clear? Is this email polite? Is this answer helpful without being harmful? There is no program that can score these, yet any person can tell a good answer from a bad one.

The obvious solution is to let people give the reward. But PPO needs a reward for every response it generates, and a single training run generates millions of them. No team of people can read and score that many responses, let alone fast enough to keep up with training.

Reinforcement Learning from Human Feedback (RLHF) solves this in two steps. First, it collects a limited amount of human feedback and trains a model to imitate it: a reward model that predicts how a person would judge a response. Then it uses this reward model in place of the person, and trains the language model on it with PPO. The idea was introduced for games and simulated robots [1], applied to language models soon after [2], [3], and became widely known with InstructGPT [4], the direct predecessor of ChatGPT.

In this post, we go through RLHF one problem at a time. We first see why people are asked to compare responses instead of scoring them, and how the Bradley-Terry model turns those comparisons into a reward model. We then see what goes wrong when PPO maximizes this reward directly, a problem called reward hacking, and how a KL penalty against a reference model prevents it. Finally, we put everything together into the full RLHF pipeline and look at its limitations.

The Three Stages of RLHF

RLHF, as used in InstructGPT [4], has three stages (Figure 1). The first two stages each use a different kind of human data, the third needs only prompts, and each stage starts from the model produced by the previous one.

1. Supervised fine-tuning (SFT). People write good answers to a set of prompts, and the pre-trained model is fine-tuned to imitate them, exactly as in Part 1. The result, ${\pi }_{\text{SFT}}$, already follows instructions reasonably well.
2. Reward model. People compare answers written by the SFT model, and a reward model $r_{\psi }$ is trained to predict their judgments.
3. RL with PPO. The policy ${\pi }_{\theta }$, initialized from ${\pi }_{\text{SFT}}$, is trained with PPO to maximize the reward model’s score.

We now go through the stages in order.

Stage 1: Supervised Fine-Tuning

The first stage is the SFT we already know from Part 1. People write good answers to a set of prompts, and the pre-trained model is trained to imitate them. In InstructGPT, this dataset had about $13,000$ prompts [4].

RLHF needs this stage for two reasons. First, the later stages work with responses written by the model itself. A pre-trained model that only continues text would mostly write responses that are all bad in the same way, and comparing them would teach us little. After SFT, the model already answers in the right format, so its responses are worth comparing. Second, the SFT model is the starting point for everything that follows: the reward model and the policy are both initialized from it, and a frozen copy of it becomes the reference model in Stage 3.

But SFT can only copy its demonstrations, and writing a demonstration for every kind of prompt is slow. Judging answers is much easier than writing them. This is what the next stage uses.

Stage 2: Learning a Reward from Human Preferences

The goal of this stage is a reward model: a model that reads a prompt and a response and outputs a number saying how much a person would like the response. The natural way to train it would be to ask people for exactly that number, a grade for each response. But for a person, giving an answer an absolute grade is hard, while deciding which of two answers is better is easy. So RLHF collects comparisons instead of grades, turns them into scores with the Bradley-Terry model, and fits the reward model to those scores. We first see why comparisons work better.

Why Compare Instead of Score

When people grade a single response from $1$ to $10$, different people use the scale differently, and even the same person scores inconsistently, so the grades end up full of noise. When they compare two responses to the same prompt instead, they agree far more, because the question does not depend on a personal scale [1], [3].

InstructGPT made the collection more efficient [4]. Instead of showing a person only two responses at a time, the SFT model writes between $4$ and $9$ responses to the same prompt, and the person ranks all of them from best to worst. A ranking contains every pairwise comparison between its responses: if A is ranked above B, then A is preferred over B. With $K$ responses there are $\left(\genfrac{}{}{0ex}{}{K}{2}\right)$ such pairs, so ranking $4$ responses gives $6$ comparisons from a single reading.

Each comparison is stored as three pieces: the prompt $x$, the response that was preferred, and the response that was not. We call the preferred response $y_w$, for “winner”, and the other one $y_l$, for “loser”. Collecting many of these gives a preference dataset of $N$ comparisons:

$$\mathcal{D}=\{(x^{(k)},y_w^{(k)},y_l^{(k)}){\}}_{k=1}^N\text{\hspace{1em}(1)}$$

where the superscript $k$ numbers the comparisons. But PPO cannot learn from a comparison directly. It needs a number for every response it generates. So the next question is how to turn “$y_w$ is better than $y_l$” into a score for each response.

From Comparisons to Scores: The Bradley-Terry Model

The Bradley-Terry model [5] does exactly this. It was introduced in 1952 to rank players from the results of their matches, and comparing two responses is the same situation: each comparison is a match, and the preferred response is the winner.

Take two items $i$ and $j$, for example two responses to the same prompt. The model assumes that each item has a strength, a positive number $p_i>0$ that measures how good it is. The strength is hidden: we never observe it directly, only the outcomes of comparisons. The stronger an item is compared with its opponent, the more often it should win, so the model sets the probability that $i$ wins to $i$’s share of the total strength of the two:

$$P(i\succ j)=\frac{p_i}{p_i+p_j}\text{\hspace{1em}(2)}$$

where $i\succ j$ means “$i$ is preferred over $j$”.

To use this with a reward model, there is one obstacle. The strengths in Eq. (2) must be positive: a negative strength would make the probability negative or greater than $1$. But a reward model is a neural network, and the number at its output can be anything, positive or negative. We cannot force it to output only positive numbers without restricting what it can learn.

The solution is to let the network output any real number, which we call the score $r_i\in \mathbb{R}$, and to turn it into a strength with the exponential function:

$$p_i=e^{r_i},p_j=e^{r_j}\text{\hspace{1em}(3)}$$

The exponential is positive for every input, so every score gives a valid strength. It also keeps the order: a higher score always means a higher strength, so a response the reward model scores higher is also more likely to be preferred. Substituting the strengths from Eq. (3) into Eq. (2):

$$P(i\succ j)=\frac{e^{r_i}}{e^{r_i}+e^{r_j}}\text{\hspace{1em}(4)}$$

Dividing the numerator and the denominator by $e^{r_i}$:

$$P(i\succ j)=\frac{\frac{e^{r_i}}{e^{r_i}}}{\frac{e^{r_i}}{e^{r_i}}+\frac{e^{r_j}}{e^{r_i}}}=\frac{1}{1+e^{r_j-r_i}}=\frac{1}{1+e^{-(r_i-r_j)}}$$

The last expression is the logistic sigmoid function,

$$\sigma (z)=\frac{1}{1+e^{-z}}$$

applied to the difference of the scores, $z=r_i-r_j$:

$$P(i\succ j)=\sigma (r_i-r_j)\text{\hspace{1em}(5)}$$

So the probability of a preference depends only on the difference between the two scores, not on the scores themselves. One consequence is that the scores have no fixed zero point. If we add the same constant $c$ to every score, every difference stays the same,

$$(r_i+c)-(r_j+c)=r_i-r_j$$

and so does every probability. A trained reward model can therefore put its scores anywhere on the number line, and only their differences carry meaning. In practice, the scores are shifted after training so that their average is $0$ [4].

The score difference also has a direct meaning, which we can see through the odds of a preference. The odds compare the chance that something happens with the chance that it does not. If $i$ is preferred $75\%$ of the time and $j$ $25\%$ of the time, the odds for $i$ are $0.75/0.25=3$, or “3 to 1”.

To compute the odds, we need both probabilities. The probability that $i$ wins is Eq. (4). The probability that $j$ wins is the same expression with $i$ and $j$ swapped, so only the numerator changes:

$$P(i\succ j)=\frac{e^{r_i}}{e^{r_i}+e^{r_j}},P(j\succ i)=\frac{e^{r_j}}{e^{r_i}+e^{r_j}}$$

Dividing the first by the second, the two denominators are the same, so they cancel. What remains is a ratio of two exponentials, which by the rule $e^a/e^b=e^{a-b}$ is a single exponential of the difference:

$$\begin{array}{rl}\frac{P(i\succ j)}{P(j\succ i)}& =\frac{\frac{e^{r_i}}{e^{r_i}+e^{r_j}}}{\frac{e^{r_j}}{e^{r_i}+e^{r_j}}}\\ & =\frac{e^{r_i}}{e^{r_j}}\\ & =e^{r_i-r_j}\end{array}$$

Taking the natural logarithm of both sides undoes the exponential:

$$\ln \frac{P(i\succ j)}{P(j\succ i)}=r_i-r_j$$

The left side is called the log-odds of the preference. So the difference between two scores is exactly the log-odds that one response is preferred over the other: a difference of $0$ means even odds, and every extra point of difference multiplies the odds by $e\approx 2.7$. This is the same relationship that defines logistic regression, where a linear function predicts the log-odds of an outcome. The Bradley-Terry model is a logistic regression in which the input is the difference between two scores.

Training the Reward Model

We can now train the reward model. It is a neural network $r_{\psi }(x,y)$ with parameters $\psi$ that reads a prompt $x$ and a response $y$ and outputs a score. We want it to give the preferred responses in the dataset $\mathcal{D}$ from Eq. (1) higher scores than the rejected ones.

In a comparison from the dataset, the two items of the Bradley-Terry model are the two responses: $i$ is the preferred response $y_w$, and $j$ is the rejected response $y_l$. Their scores are the reward model’s outputs, $r_{\psi }(x,y_w)$ and $r_{\psi }(x,y_l)$. Putting these into Eq. (5), the probability that the reward model assigns to the choice the person actually made is:

$$P(y_w\succ y_l\mid x)=\sigma (r_{\psi }(x,y_w)-r_{\psi }(x,y_l))$$

A good reward model should make the choices people actually made as likely as possible. This is the principle of maximum likelihood. If the $N$ comparisons were made independently, the probability of all of them together is the product of their individual probabilities, and we choose $\psi$ to maximize it:

$$\underset{\psi }{\max }\prod _{(x,y_w,y_l)\in \mathcal{D}}P(y_w\succ y_l\mid x)$$

A product of thousands of numbers below $1$ becomes too small for a computer to represent, so we take the logarithm. It turns the product into a sum, $\log (ab)=\log a+\log b$, and since it is increasing, it does not change which $\psi$ is best:

$$\underset{\psi }{\max }\sum _{(x,y_w,y_l)\in \mathcal{D}}\log P(y_w\succ y_l\mid x)$$

Dividing by $N$ turns the sum into an average over the dataset, written as an expectation, again without changing the best $\psi$. Substituting the probability from above:

$$\underset{\psi }{\max }{\mathbb{E}}_{(x,y_w,y_l)\sim \mathcal{D}}\left[\log \sigma \right(r_{\psi }(x,y_w)-r_{\psi }(x,y_l)\left)\right]$$

Neural networks are trained by minimizing a loss with gradient descent, not by maximizing. Maximizing a quantity is the same as minimizing its negative, so we flip the sign. The result is the negative log-likelihood, the reward model loss:

$${\mathcal{L}}_{\text{RM}}(\psi )=-{\mathbb{E}}_{(x,y_w,y_l)\sim \mathcal{D}}\left[\log \sigma \right(r_{\psi }(x,y_w)-r_{\psi }(x,y_l)\left)\right]\text{\hspace{1em}(6)}$$

To see how this loss trains the reward model, we take its gradient with respect to the parameters $\psi$. To keep the notation short, we write the gap between the two scores as

$$\Delta =r_{\psi }(x,y_w)-r_{\psi }(x,y_l)$$

so that Eq. (6) becomes:

$${\mathcal{L}}_{\text{RM}}(\psi )=-{\mathbb{E}}_{(x,y_w,y_l)\sim \mathcal{D}}[\log \sigma (\Delta )]$$

The gradient can be moved inside the expectation, because the dataset does not depend on $\psi$:

$${\nabla }_{\psi }{\mathcal{L}}_{\text{RM}}(\psi )=-{\mathbb{E}}_{(x,y_w,y_l)\sim \mathcal{D}}[{\nabla }_{\psi }\log \sigma (\Delta )]$$

Now we break down the gradient inside. The logarithm is applied to $\sigma$, which is applied to $\Delta$, which depends on $\psi$. By the chain rule, using $\frac{d}{dz}\log z=\frac{1}{z}$:

$${\nabla }_{\psi }\log \sigma (\Delta )=\frac{1}{\sigma (\Delta )}{\sigma }^{\prime }(\Delta ){\nabla }_{\psi }\Delta$$

The sigmoid has a convenient derivative:

$${\sigma }^{\prime }(\Delta )=\sigma (\Delta )(1-\sigma (\Delta ))$$

Substituting it, the $\sigma (\Delta )$ in the numerator and the denominator cancels:

$${\nabla }_{\psi }\log \sigma (\Delta )=\frac{\sigma (\Delta )(1-\sigma (\Delta ))}{\sigma (\Delta )}{\nabla }_{\psi }\Delta =(1-\sigma (\Delta )){\nabla }_{\psi }\Delta$$

The gradient of the gap is the difference of the gradients of the two scores:

$${\nabla }_{\psi }\Delta ={\nabla }_{\psi }{r}_{\psi }(x,y_w)-{\nabla }_{\psi }{r}_{\psi }(x,y_l)$$

Putting everything together gives the gradient of Eq. (6):

$${\nabla }_{\psi }{\mathcal{L}}_{\text{RM}}(\psi )=-{\mathbb{E}}_{(x,y_w,y_l)\sim \mathcal{D}}\left[\right(1-\sigma (\Delta )\left)\right({\nabla }_{\psi }{r}_{\psi }(x,y_w)-{\nabla }_{\psi }{r}_{\psi }(x,y_l)\left)\right]\text{\hspace{1em}(7)}$$

Gradient descent moves against this gradient, so each step adds a multiple of ${\nabla }_{\psi }{r}_{\psi }(x,y_w)$, raising the score of the preferred response, and subtracts a multiple of ${\nabla }_{\psi }{r}_{\psi }(x,y_l)$, lowering the score of the rejected one.

The size of the step is set by the factor $1-\sigma (\Delta )$. Since $\sigma (\Delta )$ is the probability the reward model gives to the person’s choice, this factor is the probability it gives to the opposite choice:

$$1-\sigma (\Delta )=1-P(y_w\succ y_l\mid x)=P(y_l\succ y_w\mid x)$$

When the model strongly disagrees with the person, this probability is close to $1$ and the update is large. When the model already agrees, it is close to $0$ and the update fades.

In architecture, the reward model is built like the value model from Part 3. It is an LLM, usually initialized from the SFT model so that it already understands language, with its language-model head replaced by a single linear layer that outputs one number. It reads the prompt followed by the full response, and the number it outputs at the last token is the score.

The two models differ in what they predict and when they are trained. The value model estimates the expected future reward from every intermediate state of a response, and it is trained during PPO, alongside the policy. The reward model scores only finished responses, and it is trained once, before PPO, and then frozen. In InstructGPT, the reward model had $6$ billion parameters and was trained on comparisons from about $33,000$ prompts [4].

Stage 3: Optimizing the Policy with PPO

With the reward model trained and frozen, the last stage trains the policy. It starts as a copy of the SFT model and is improved with PPO from Part 3, with the reward model in place of a checker. We first see why maximizing the reward model’s score directly does not work, then how a KL penalty fixes it, how that penalty is turned into the token rewards PPO needs, and finally one more term that keeps the model’s old skills.

The Naive Objective and Reward Hacking

With a reward model in hand, Stage 3 seems straightforward: generate a response $y$ for each prompt $x$ with the policy, score it with $r_{\psi }(x,y)$, and run PPO to maximize the score:

$$\underset{\theta }{\max }{\mathbb{E}}_{x\sim \mathcal{D},y\sim {\pi }_{\theta }(\cdot \mid x)}[r_{\psi }(x,y)]\text{\hspace{1em}(8)}$$

This fails, and the reason is that the reward model is not a person. It is a model that has learned to imitate human judgments on the responses it was trained on, which were all written by the SFT model. As PPO trains, the policy starts writing responses that look different from those, and on such responses the reward model’s scores are no longer reliable. PPO is very good at finding whatever the reward model rewards, including its mistakes. The policy learns to write responses that score high without being good: answers that are needlessly long because the reward model slightly prefers length [6], that flatter the user, or that repeat phrases the reward model likes. This is reward hacking, an example of Goodhart’s law: when a measure becomes a target, it ceases to be a good measure [7].

Reward hacking can be measured. Gao et al. [8] trained policies against a reward model and checked them against a much larger “gold” reward model standing in for real human judgment. As the policy moved further from its starting point, the score from the trained reward model kept rising, but the gold score first rose, then peaked, and then fell. Past that point, the policy was getting better at fooling the reward model and worse at the actual task.

The KL Penalty: Staying Close to the Reference Model

The reward model is reliable only near the responses it was trained on, that is, near the SFT model. So the fix is to keep the policy close to the SFT model. We make a frozen copy of it, the reference model ${\pi }_{\text{ref}}={\pi }_{\text{SFT}}$, and subtract a penalty from the reward that grows as the policy moves away from it. As in Part 2, we measure this distance with the KL divergence [2], [4]:

$$\underset{\theta }{\max }{\mathbb{E}}_{x\sim \mathcal{D},y\sim {\pi }_{\theta }(\cdot \mid x)}[r_{\psi }(x,y)]-\beta {\mathbb{E}}_{x\sim \mathcal{D}}\left[D_{\text{KL}}\right({\pi }_{\theta }(\cdot \mid x)\Vert {\pi }_{\text{ref}}(\cdot \mid x)\left)\right]\text{\hspace{1em}(9)}$$

The coefficient $\beta$ sets the balance. With $\beta =0$, we are back to the naive objective in Eq. (8) and to reward hacking. With a very large $\beta$, the policy cannot move at all and stays the SFT model. In between, the policy can improve the reward only as long as the improvement is worth the distance it costs.

This KL term looks like TRPO’s constraint from Eq. (7) in Part 2, but it plays a different role. TRPO’s KL is measured against ${\pi }_{{\theta }_{\text{old}}}$, the model from the start of the current iteration, and limits the size of each update. Within PPO, clipping already does that job. The KL in Eq. (9) is measured against ${\pi }_{\text{ref}}$, the model from the start of training, and limits how far the policy can move in total, however many small, safe updates it takes to get there. So RLHF needs both: clipping keeps every single update small, and the KL penalty keeps the sum of all the updates from carrying the policy too far from the SFT model.

From a Sequence Penalty to Token Rewards

PPO, as we saw in Part 3, works with one reward $r_t$ per token. The reward model gives one score per response, and the KL in Eq. (9) is also defined over whole responses. To use PPO, we need to split both into token rewards.

The KL divergence between the two models, for a prompt $x$, is the expected log-ratio of the probabilities they give to a response:

$$D_{\text{KL}}({\pi }_{\theta }(\cdot \mid x)\Vert {\pi }_{\text{ref}}(\cdot \mid x))={\mathbb{E}}_{y\sim {\pi }_{\theta }(\cdot \mid x)}\left[\log \frac{{\pi }_{\theta }(y\mid x)}{{\pi }_{\text{ref}}(y\mid x)}\right]$$

A language model gives the probability of a response as the product of its token probabilities, where $s_t$ is the prompt and the tokens before $y_t$:

$$\pi (y\mid x)=\prod _{t=0}^T\pi (y_t\mid s_t)$$

The logarithm turns the product into a sum, so the log-ratio of a response splits into one log-ratio per token:

$$\log \frac{{\pi }_{\theta }(y\mid x)}{{\pi }_{\text{ref}}(y\mid x)}=\sum _{t=0}^T\log \frac{{\pi }_{\theta }(y_t\mid s_t)}{{\pi }_{\text{ref}}(y_t\mid s_t)}\text{\hspace{1em}(10)}$$

Now we put Eq. (10) back into Eq. (9). The KL term becomes an expectation over responses sampled from the policy, the same expectation as the reward term, so the two can be merged into one. Then the sum over tokens can be pulled out:

$$\begin{array}{rl}& {\mathbb{E}}_{x\sim \mathcal{D},y\sim {\pi }_{\theta }(\cdot \mid x)}\left[r_{\psi }(x,y)-\beta \log \frac{{\pi }_{\theta }(y\mid x)}{{\pi }_{\text{ref}}(y\mid x)}\right]\\ =& {\mathbb{E}}_{x\sim \mathcal{D},y\sim {\pi }_{\theta }(\cdot \mid x)}\left[r_{\psi }(x,y)-\beta \sum _{t=0}^T\log \frac{{\pi }_{\theta }(y_t\mid s_t)}{{\pi }_{\text{ref}}(y_t\mid s_t)}\right]\end{array}$$

Inside the expectation, we now have one penalty term per token and one score for the whole response. We can assign them to tokens: every token gets its own penalty, which grows with how much more likely the policy makes that token than the reference model does, and the last token, $t=T$, also gets the reward model’s score, since that is when the response is finished [2], [9]:

$$r_t=-\beta \log \frac{{\pi }_{\theta }(y_t\mid s_t)}{{\pi }_{\text{ref}}(y_t\mid s_t)}+\{\begin{array}{ll}{r}_{\psi }(x,y)& \text{if }t=T\\ 0& \text{otherwise}\end{array}\text{\hspace{1em}(11)}$$

Now we check that these token rewards give back the objective. For short, we write the log-ratio of token $t$ as

$${\ell }_t=\log \frac{{\pi }_{\theta }(y_t\mid s_t)}{{\pi }_{\text{ref}}(y_t\mid s_t)}$$

Then we expand the sum inside the expectation, move the reward model’s score next to the last token’s penalty, and recognize each group as a token reward from Eq. (11):

$$\begin{array}{rl}{r}_{\psi }(x,y)-\beta \sum _{t=0}^T{\ell }_t& =r_{\psi }(x,y)-\beta {\ell }_0-\beta {\ell }_1-\cdots -\beta {\ell }_T\\ & =(-\beta {\ell }_0)+(-\beta {\ell }_1)+\cdots +(-\beta {\ell }_T+r_{\psi }(x,y))\\ & =r_0+r_1+\cdots +r_T\\ & =\sum _{t=0}^T{r}_t\end{array}$$

Putting this back inside the expectation, and flipping the sign to turn the objective into a loss to minimize, the total loss of Stage 3, with the KL penalty, is:

$${\mathcal{L}}^{\text{RLHF}}(\theta )=-{\mathbb{E}}_{x\sim \mathcal{D},y\sim {\pi }_{\theta }(\cdot \mid x)}\left[\sum _{t=0}^T{r}_t\right]\text{\hspace{1em}(12)}$$

This sum is something PPO already knows. In Part 3, we defined the return $G_t$ as the sum of all the rewards from step $t$ to the end of the response, each discounted by $\gamma$ for every step it lies ahead (Eq. (3) in Part 3). For the first token, $t=0$, and with $\gamma =1$ as is common for LLMs, the discounting disappears and the return is exactly our sum:

$$G_0=r_0+\gamma r_1+\cdots +{\gamma }^T{r}_T=r_0+r_1+\cdots +r_T=\sum _{t=0}^T{r}_t$$

So minimizing the loss in Eq. (12) is the same as maximizing the expected return, which is exactly what PPO does.

PPO does not minimize Eq. (12) directly. Instead, it computes an advantage ${\hat{A}}_t$ for every token from the token rewards of Eq. (11), with the value model and GAE, and maximizes the clipped objective from Eq. (11) in Part 3. This is what is actually optimized at each update:

$$L^{\text{CLIP}}(\theta )={\mathbb{E}}_t\left[\min \right({\rho }_t(\theta ){\hat{A}}_t,\text{clip}({\rho }_t(\theta ),1-ϵ,1+ϵ){\hat{A}}_t\left)\right]\text{\hspace{1em}(13)}$$

The reward model and the KL penalty do not appear in this equation directly. They enter through the advantages ${\hat{A}}_t$, which are computed from the token rewards. Note also that the ratio ${\rho }_t$ here is not the log-ratio in the KL penalty. The ratio compares the policy with ${\pi }_{{\theta }_{\text{old}}}$, the policy that generated the batch, and is clipped to keep each update small. The log-ratio in the penalty compares the policy with the frozen reference model, and it is computed once when the batch is generated and then kept fixed as part of the reward, so it is never clipped.

RLHF changes only the reward. Everything else in Stage 3 is PPO exactly as in Part 3.

Each token’s reward $r_t$ now comes from Eq. (11): the KL penalty for every token, plus the reward model’s score at the last one. From these rewards on, nothing is new. The value model $V_{\varphi }$ is trained to predict them with the value loss, the TD errors and the GAE advantages are computed from them, and the policy is updated with the probability ratio and clipping, over several epochs of each batch. Stage 3 uses four models: the policy ${\pi }_{\theta }$, the value model $V_{\varphi }$, the reward model $r_{\psi }$, and the reference model ${\pi }_{\text{ref}}$. Of these, only the policy and the value model are updated. The reward model was trained in Stage 2 and stays frozen, and so does the reference model.

Keeping Old Skills: The Pretraining Mix

The KL penalty keeps the policy’s responses close to the reference model, but RLHF still has a side effect. InstructGPT found that after PPO, the model performed worse on several standard NLP benchmarks than before, a cost the authors called the alignment tax [4]. The reason is that PPO only rewards what the reward model cares about, and skills that the reward model does not measure can slowly fade.

To reduce this, InstructGPT added a second term to the clipped objective in Eq. (13): the ordinary language-modeling log-likelihood, the same objective as in pre-training, computed on text $x$ drawn from the pre-training data ${\mathcal{D}}_{\text{pretrain}}$ and weighted by a coefficient $\eta$:

$$L^{\text{PPO-ptx}}(\theta )=L^{\text{CLIP}}(\theta )+\eta {\mathbb{E}}_{x\sim {\mathcal{D}}_{\text{pretrain}}}[\log {\pi }_{\theta }(x)]\text{\hspace{1em}(14)}$$

The new term needs no RL at all: it is ordinary next-token prediction on fixed text, so its gradient is simply added to PPO’s. It keeps training the model on the pre-training task, so the model does not forget what it learned there. InstructGPT called this variant PPO-ptx [4].

The Full RLHF Pipeline

Putting the three stages together, with the PPO loop from Part 3 inside Stage 3:

This is where the four models from Part 3, the policy, the value model, the reward model, and the reference model, come from. The policy and the value model belong to PPO and are trained. The reward model comes from Stage 2, and the reference model is the frozen SFT model from Stage 1. Both are used only for forward passes.

The results showed how well this works. In InstructGPT, people preferred the answers of a $1.3$-billion-parameter RLHF model over those of the original $175$-billion-parameter GPT-3, a model more than $100$ times larger [4]. The RLHF model also made up facts less often and produced less toxic text. Anthropic later used the same recipe to train assistants that are both helpful and harmless [10].

Limitations of RLHF

RLHF made language models far more useful, but it has clear limitations [11]:

• It is expensive. Stage 3 needs four LLMs in memory at the same time (the policy, the value model, the reward model, and the reference model), and PPO generates new responses at every iteration. Human comparisons are also slow and costly to collect.
• The reward model is a weak point. It is only an approximation of human judgment, trained on a limited dataset. It inherits the biases of its labelers and of its data, such as a preference for longer answers [6], and the policy can still exploit its mistakes if $\beta$ is too small.
• It is complex and unstable. RLHF combines two separately trained models and an RL algorithm with many hyperparameters, and small implementation details can change the result [9].

Most of these problems come from the same place: the reward model and the RL loop that optimizes it. The reward model learns from preferences, and then PPO learns from the reward model. This raises a natural question.

If the preferences are what we actually care about, do we need the reward model and the RL loop at all?

Direct Preference Optimization shows that we do not, and it is the topic of Part 5.

References

1. Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., & Amodei, D. (2017). Deep Reinforcement Learning from Human Preferences. NeurIPS.
2. Ziegler, D. M., Stiennon, N., Wu, J., et al. (2019). Fine-Tuning Language Models from Human Preferences. arXiv:1909.08593.
3. Stiennon, N., Ouyang, L., Wu, J., et al. (2020). Learning to Summarize from Human Feedback. NeurIPS.
4. Ouyang, L., Wu, J., Jiang, X., et al. (2022). Training Language Models to Follow Instructions with Human Feedback. NeurIPS.
5. Bradley, R. A., & Terry, M. E. (1952). Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, 39(3/4), 324–345.
6. Singhal, P., Goyal, T., Xu, J., & Durrett, G. (2023). A Long Way to Go: Investigating Length Correlations in RLHF. arXiv:2310.03716.
7. Manheim, D., & Garrabrant, S. (2018). Categorizing Variants of Goodhart’s Law. arXiv:1803.04585.
8. Gao, L., Schulman, J., & Hilton, J. (2023). Scaling Laws for Reward Model Overoptimization. ICML.
9. Huang, S., Noukhovitch, M., Hosseini, A., Rasul, K., Wang, W., & Tunstall, L. (2024). The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization. arXiv:2403.17031.
10. Bai, Y., Jones, A., Ndousse, K., et al. (2022). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv:2204.05862.
11. Casper, S., Davies, X., Shi, C., et al. (2023). Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback. TMLR.