# Buy RLHF and preference data for model alignment

URL: https://sellmydatatoai.net/buy-ai-training-data/rlhf-preference-data

RLHF and preference data are human judgments on model outputs, usually pairs of responses with one marked as better, used to train reward models and align models with human preferences. OpenAI's 2022 InstructGPT paper showed the effect: outputs from a 1.3 billion parameter model trained this way were preferred to those of the 175 billion parameter GPT-3.

## Key takeaways

- AI teams use rlhf and preference data for (1) reward model training, (2) reinforcement learning from human feedback and (3) direct preference optimization and similar methods.
- Usually priced per comparison or per rated prompt.
- Common delivery formats: JSONL with prompt, chosen and rejected responses, ranked lists of several responses per prompt and rubric scores with rater IDs and rationales.
- Check before you license rlhf and preference data: (1) rater qualifications and guidelines supplied, (2) agreement between raters measured and (3) model that produced the responses and its terms.
- Every seller we shortlist passes 8 vetting checks before you see a sample.

## Buyer's spec

- Used for: Reward model training; Reinforcement learning from human feedback; Direct preference optimization and similar methods; Safety tuning with labeled red-team data
- Formats: JSONL with prompt, chosen and rejected responses; Ranked lists of several responses per prompt; Rubric scores with rater IDs and rationales; Rater guidelines and calibration records
- Usually priced: Per comparison or per rated prompt
- Check before you sign: Rater qualifications and guidelines supplied; Agreement between raters measured; Model that produced the responses and its terms; Rater consent and pay records; Offensive content flagged and labeled; No duplication with public preference datasets

## What is RLHF and preference data?

Preference data is a record of human choices between model outputs: for a given prompt, which response is better, or how a set of responses ranks. RLHF, reinforcement learning from human feedback, uses those choices to train a reward model and then tunes the main model against it. The glossary covers [RLHF](https://sellmydatatoai.net/glossary/rlhf) and [preference data](https://sellmydatatoai.net/glossary/preference-data) in brief.

The InstructGPT paper by Long Ouyang and colleagues at OpenAI, published in March 2022, describes the standard recipe: collect labeler demonstrations for [supervised fine-tuning](https://sellmydatatoai.net/glossary/supervised-fine-tuning), then collect rankings of model outputs to fine-tune further with RLHF.

## What does a preference dataset look like?

A preference dataset is usually a file of prompts with paired responses. Anthropic's public HH-RLHF dataset on Hugging Face is a clear example: each line of its JSONL files holds a pair of texts, one chosen and one rejected, and it is released under the MIT license.

The same dataset card shows why documentation matters. Anthropic states that the preference data is meant to train preference or reward models and is not meant for supervised training of dialogue agents, because training dialogue agents on it is likely to lead to harmful models.

| Format | What raters do | Typical use |
| --- | --- | --- |
| Pairwise | Pick the better of two responses | Reward models |
| Ranked | Order several responses to one prompt | Reward models with more comparisons per prompt |
| Rubric scored | Score responses against written criteria | Evaluation and targeted tuning |
| Edited | Rewrite a response into a better one | Supervised fine-tuning |

## What makes preference data good quality?

Good preference data comes from qualified raters following written guidelines, with agreement between raters measured. For expert domains such as law or medicine, the rater's credentials matter as much as the volume of judgments.

Ask for the rater guidelines, a calibration record and agreement scores. Low agreement is not always a fault, since some prompts are genuinely contested, but the seller should know which ones and flag them.

## How is preference data priced?

Preference data is priced per comparison or per rated prompt. No buyer publishes a rate card, so treat any quote as a starting point and compare it with the cost of commissioning the same work from raters you recruit yourself.

Our guide to [AI training data pricing models](https://sellmydatatoai.net/guides/ai-training-data-pricing-models) sets out the common structures, including per-unit pricing and fixed project fees.

## Who supplies preference data?

Preference data is supplied by training-data companies that run rater pools, by AI product companies that collect user ratings, and by firms whose experts review model output as part of their work. Profiles of training-data companies such as [Scale AI](https://sellmydatatoai.net/data-buyers/scale-ai) and [Mercor](https://sellmydatatoai.net/data-buyers/mercor) are on our data buyers pages.

User ratings from AI products, such as thumbs up and thumbs down on responses, are preference data too. They are less controlled than rater judgments and usually tied to user accounts, so they need anonymization and a privacy notice that covers the use before licensing.

## How should preference data be delivered?

Preference data should be delivered as JSONL with one comparison per line, including the prompt, each response, the rater's choice, a rater ID, a timestamp and any written rationale. Rater guidelines should come as a versioned document, since a change to the guidelines changes what the labels mean.

## What should you check before buying preference data?

Before buying preference data, check which model generated the responses being compared and that its terms allow its outputs to be used this way. Confirm that raters consented and were paid, and that offensive content from red-teaming is labeled as such.

Check a random sample for duplication with public sets such as HH-RLHF. [Book a free consultation](https://sellmydatatoai.net/book-a-consultation?side=buyer) to source vetted preference data, and use our [due diligence checklist](https://sellmydatatoai.net/guides/ai-training-data-due-diligence-checklist) for the contract.

## Frequently asked questions

### What is the difference between RLHF data and fine-tuning data?

RLHF data records which of several responses a person preferred, while fine-tuning data shows the single response a model should give. The InstructGPT paper used both: labeler demonstrations for supervised fine-tuning, then rankings of model outputs to train a reward model for reinforcement learning.

### Can preference data be used for supervised training?

Preference data is not always suitable for supervised training. Anthropic's HH-RLHF dataset card states that its data is meant for preference or reward models and not for supervised training of dialogue agents, because that is likely to produce harmful models. Check the intended use stated by any seller.

### Who should rate responses in a preference dataset?

Responses in a preference dataset should be rated by people qualified in the subject, following written guidelines. General raters suit everyday helpfulness tasks, while legal, medical or coding prompts need practitioners. Ask the seller for rater credentials, guidelines and agreement scores before you buy.

### Are public preference datasets free to use commercially?

Public preference datasets carry their own licenses, so commercial use depends on each one. Anthropic's HH-RLHF is released under the MIT license, for example, but other sets restrict commercial use. Licensed private preference data gives you judgments that competitors have not trained on.

## Sources

- [Training language models to follow instructions with human feedback](https://arxiv.org/abs/2203.02155), arXiv (Ouyang et al., 2022), accessed November 2026
- [Anthropic/hh-rlhf dataset card](https://huggingface.co/datasets/Anthropic/hh-rlhf), Hugging Face, accessed November 2026

---

Book a free consultation: https://sellmydatatoai.net/book-a-consultation