# Preference data

URL: https://sellmydatatoai.net/glossary/preference-data

Preference data records which of two or more model outputs a person judged better for a given prompt, and is used to train reward models or tune models directly.

Preference data is a set of records in which a person, or sometimes a model, has judged which of two or more responses to the same prompt is better. Each record usually holds the prompt, a chosen response, a rejected response and optional ratings or notes.

## What is preference data used for?

Preference data is used to train reward models for [RLHF](https://sellmydatatoai.net/glossary/rlhf) and to tune models directly with methods such as direct preference optimization (DPO), introduced by Rafael Rafailov and colleagues in 2023. It teaches qualities that are hard to write down as a single correct answer, such as helpfulness, tone and safety.

## What makes preference data valuable?

Preference data is most valuable when raters are domain experts and their agreement is measured. Buyers check rater instructions, inter-rater agreement, the share of ties and whether prompts reflect real use.

AI product companies may already hold preference signals in user ratings and side-by-side feedback. See [RLHF and preference data](https://sellmydatatoai.net/buy-ai-training-data/rlhf-preference-data) and [evaluation dataset](https://sellmydatatoai.net/glossary/evaluation-dataset).