Training language models to follow instructions with human feedback

Hello, lets now start our RL for language modeling paper series. The main goal of this paper is to align large language models with user intent by fine tuning them on a user feedback. This aims to reduce the generation of harmful, untruthful, toxic and not helpful outputs. They fine tune GPT-3 on a dataset containing demonstration of preferred user outputs for supervised training and then further rank the model outputs to do reinforcement learning from human feedback(RLHF).

We can summarize the main steps as:

1. Fine turning model on demonstration using SFT with the goal of demonstrating the desired behavior.
2. Generate several outputs and rank those outputs which will be used to train a reward model.
3. Use PPO to train the policy based on the reward it gets from our reward model on the outputs it generated and they call this model instructGPT.

While training the reward model, to speed up comparisons they show the labelers anywhere between 4 to 9 options and ask them to choose 2 and use Bradley-Terry model like binary cross entropy to calculate the loss.

• the (k,2) is the number of ways to choose 2 elements from k elements where order doesn’t matter.
• D is the dataset collected from human
• r is the reward given for the output y for given prompt x.
• the yw is the human preferred output and yl is the model generated output.

So the main goal is minimizing the expected log loss between the human preference and the model generation.

Here is the final objective we’re opining towards when we do our PPO. First the PPO Objective

• We’re sampling prompt x and generate response y under our rl policy(the model we’re finetuning).
• For our generation our reward model will give us a scalar reward r.
• −βlog⁡(πϕRL(y∣x)/πSFT(y∣x)) part is the per token KL penalty which prevent the rl policy highly drift from the SFT model.

The second part is the Pretraining objective.

• γlog⁡(πϕRL(x) - is trying to maximize the likelihood of our RL policy to generate outputs like the pretrained model.

Main Findings

• The 1.3B parameter instructGPT finetuned on this human preference data generate outputs much preferred to the 175B parameter instructGPT which wasn’t, despite of having same architecture and more parameters.
• This model score twice on Truthful QA benchmark generating truthful and informative answers and has half hallucination rate(making up information that wasn’t in the input) than GPT-3.
• This model generate 25% less toxic output than GPT-3, but not less biased.
• The model can still make simple mistakes as failing to follow instructions, making up facts.

References

Ouyang, Long, et al. “Training language models to follow instructions with human feedback.” Advances in neural information processing systems 35 (2022): 27730-27744.