# RLHF

URL: https://sellmydatatoai.net/glossary/rlhf

RLHF (reinforcement learning from human feedback) trains a model on human judgments of its outputs, usually through a reward model learned from preference comparisons.

RLHF, short for reinforcement learning from human feedback, is a training method in which people compare or rate a model's outputs, a reward model learns to predict those judgments, and the main model is then optimized with reinforcement learning to score well against that reward model.

## How does RLHF work?

1. **Collect comparisons**: Human raters see two or more responses to the same prompt and pick the better one.
2. **Train a reward model**: A separate model learns to predict which response raters prefer.
3. **Optimize the policy**: The language model is trained with reinforcement learning, often proximal policy optimization (PPO), to maximize the reward, with a penalty for drifting too far from the original model.

The approach was described by [Paul Christiano and colleagues in 2017](https://arxiv.org/abs/1706.03741) and applied to instruction following in OpenAI's [InstructGPT paper](https://arxiv.org/abs/2203.02155) in 2022. [Direct preference optimization](https://arxiv.org/abs/2305.18290) (DPO), introduced in 2023, uses the same [preference data](https://sellmydatatoai.net/glossary/preference-data) without a separate reward model.

RLHF usually follows [supervised fine-tuning](https://sellmydatatoai.net/glossary/supervised-fine-tuning). AI teams sourcing this data can see [RLHF and preference data](https://sellmydatatoai.net/buy-ai-training-data/rlhf-preference-data) or [submit a data requirement](https://sellmydatatoai.net/request-data).