RLHF
By Gareth HoyleLast updated How we research
RLHF (reinforcement learning from human feedback) trains a model on human judgments of its outputs, usually through a reward model learned from preference comparisons.
RLHF, short for reinforcement learning from human feedback, is a training method in which people compare or rate a model's outputs, a reward model learns to predict those judgments, and the main model is then optimized with reinforcement learning to score well against that reward model.
How does RLHF work?
- 1
Collect comparisons
Human raters see two or more responses to the same prompt and pick the better one.
- 2
Train a reward model
A separate model learns to predict which response raters prefer.
- 3
Optimize the policy
The language model is trained with reinforcement learning, often proximal policy optimization (PPO), to maximize the reward, with a penalty for drifting too far from the original model.
The approach was described by Paul Christiano and colleagues in 2017 and applied to instruction following in OpenAI's InstructGPT paper in 2022. Direct preference optimization (DPO), introduced in 2023, uses the same preference data without a separate reward model.
RLHF usually follows supervised fine-tuning. AI teams sourcing this data can see RLHF and preference data or submit a data requirement.