Buy RLHF and preference data for model alignment

What is RLHF and preference data?

Preference data is a record of human choices between model outputs: for a given prompt, which response is better, or how a set of responses ranks. RLHF, reinforcement learning from human feedback, uses those choices to train a reward model and then tunes the main model against it. The glossary covers RLHF and preference data in brief.

The InstructGPT paper by Long Ouyang and colleagues at OpenAI, published in March 2022, describes the standard recipe: collect labeler demonstrations for supervised fine-tuning, then collect rankings of model outputs to fine-tune further with RLHF.

What does a preference dataset look like?

A preference dataset is usually a file of prompts with paired responses. Anthropic's public HH-RLHF dataset on Hugging Face is a clear example: each line of its JSONL files holds a pair of texts, one chosen and one rejected, and it is released under the MIT license.

The same dataset card shows why documentation matters. Anthropic states that the preference data is meant to train preference or reward models and is not meant for supervised training of dialogue agents, because training dialogue agents on it is likely to lead to harmful models.

What makes preference data good quality?

Good preference data comes from qualified raters following written guidelines, with agreement between raters measured. For expert domains such as law or medicine, the rater's credentials matter as much as the volume of judgments.

Ask for the rater guidelines, a calibration record and agreement scores. Low agreement is not always a fault, since some prompts are genuinely contested, but the seller should know which ones and flag them.

Preference data is priced per comparison or per rated prompt. No buyer publishes a rate card, so treat any quote as a starting point and compare it with the cost of commissioning the same work from raters you recruit yourself.

Our guide to AI training data pricing models sets out the common structures, including per-unit pricing and fixed project fees.

Who supplies preference data?

Preference data is supplied by training-data companies that run rater pools, by AI product companies that collect user ratings, and by firms whose experts review model output as part of their work. Profiles of training-data companies such as Scale AI and Mercor are on our data buyers pages.

User ratings from AI products, such as thumbs up and thumbs down on responses, are preference data too. They are less controlled than rater judgments and usually tied to user accounts, so they need anonymization and a privacy notice that covers the use before licensing.

How should preference data be delivered?

Preference data should be delivered as JSONL with one comparison per line, including the prompt, each response, the rater's choice, a rater ID, a timestamp and any written rationale. Rater guidelines should come as a versioned document, since a change to the guidelines changes what the labels mean.

What should you check before buying preference data?

Before buying preference data, check which model generated the responses being compared and that its terms allow its outputs to be used this way. Confirm that raters consented and were paid, and that offensive content from red-teaming is labeled as such.

Check a random sample for duplication with public sets such as HH-RLHF. Book a free consultation to source vetted preference data, and use our due diligence checklist for the contract.