# Sourcing expert and human-generated data: vendors, costs and quality

URL: https://sellmydatatoai.net/guides/sourcing-expert-and-human-data
Reviewed: 2026-11-08

Expert and human-generated data is commissioned work, such as demonstrations, preference comparisons and red-team attacks, bought from vendors including Scale AI, Surge AI, Mercor, Turing, micro1, Invisible and Handshake AI. None of them publishes a buyer rate card, so price is set per hour, task or project. Quality depends on rater screening, calibration and overlap checks you can audit.

## Key takeaways

- Human data covers demonstrations for fine-tuning, preference comparisons for RLHF, red teaming and evaluation tasks.
- None of the seven vendors in this guide publishes buyer pricing, but contributor pay gives a floor: Mercor shows a $90 an hour average contracted rate.
- OpenAI's InstructGPT paper reported agreement of 72.6% between its training labelers, which is a useful reference point for preference data.
- Ask every vendor for screening tests, gold tasks, overlap rates and versioned guidelines, and audit a random sample yourself.
- Your contract needs IP assignments from every contributor and rules on reuse of your prompts with other clients.

Licensed archives tell a model what people have already written. Expert and human-generated data is made to order: people write the ideal answer, compare two model outputs or try to make a model fail. Labs commission it because it targets the exact behavior they want to change.

This guide covers the main types, the vendors that sell them, how pricing works and how to control quality. Every vendor detail below comes from the vendor's own site, checked in November 2026.

## What types of expert and human data do AI teams buy?

AI teams buy four main types of human data: demonstrations, preference comparisons, red-team attacks and evaluation tasks. The InstructGPT paper by Long Ouyang and colleagues at OpenAI set the template in 2022, using about 40 contractors hired on Upwork and through Scale AI to write demonstrations and rank model outputs.

| Type | What people do | Used for | Usual pricing basis |
| --- | --- | --- | --- |
| Demonstrations | Write the ideal response to a prompt | Supervised fine-tuning | Per task or per hour |
| Preference data | Compare or rank model outputs | Reward models and RLHF | Per comparison or per hour |
| Red teaming | Try to make a model produce harmful output | Safety tuning and testing | Per hour or per project |
| Evaluation tasks | Write tasks with reference answers and rubrics | Measuring capability | Per task or per set |
| Agent trajectories and environments | Complete or design multi-step tasks | Agent training and grading | Per environment or per project |

Our pages on [RLHF and preference data](https://sellmydatatoai.net/buy-ai-training-data/rlhf-preference-data) and [agent trajectories and RL environments](https://sellmydatatoai.net/guides/agent-trajectories-and-rl-environments) cover two of these in depth.

## Which vendors supply expert human data?

This guide covers seven vendors that sell expert human data to AI labs: Scale AI, Surge AI, Mercor, Turing, micro1, Invisible and Handshake AI. The table records what each states on its own site, which is marketing copy, not verified performance.

| Vendor | What it says it offers | Published figures | Pricing published |
| --- | --- | --- | --- |
| Scale AI | Generative AI Data Engine with datasets curated by vetted subject matter experts, targeted red teaming and an Ops Center for quality control | None on the product page | No |
| Surge AI | Off-the-shelf data, a training lab and an expert workforce, plus its own benchmarks such as DAYJOB: Tax | None on the homepage | No |
| Mercor | Human evaluation, expert data, benchmarks and RL environments built on professional work | 5M+ experts, $90 an hour average contracted rate | No |
| Turing | Datasets, RL environments and expert talent for frontier labs in software engineering, enterprise knowledge work and frontier STEM | None | No |
| micro1 | Expert data and evaluation environments, with Zara, an AI recruiter agent that sources and vets experts | 100+ fields | No |
| Invisible | Training data and RL environments, with Meridial, a network of vetted experts | Says it has trained over 80% of the world's leading AI models | No |
| Handshake AI | Expert data on demand and RL environments for frontier labs and enterprises | 30M+ experts in its network, 200+ in-house researchers and engineers | No |

Several of these companies also buy existing company data through separate programs. Our profiles of [Scale AI](https://sellmydatatoai.net/data-buyers/scale-ai), [Mercor](https://sellmydatatoai.net/data-buyers/mercor), [micro1](https://sellmydatatoai.net/data-buyers/micro1) and [Handshake AI](https://sellmydatatoai.net/data-buyers/handshake-ai) cover those programs from the seller's side.

## How much does expert human data cost?

Expert human data is priced per hour, per task or per project, and none of the seven vendors above publishes a buyer rate card. The pay rates vendors publish to contributors give a floor, because the buyer price must also cover recruiting, review, tooling and margin.

| Vendor | Published contributor pay | Context |
| --- | --- | --- |
| Mercor | $90 an hour average contracted rate | Homepage figure across its expert network |
| Mercor | $60 to $70 an hour | Sales and marketing expert role listing |
| Mercor | $65 to $75 | Chemicals safety for red teaming role listing |
| Handshake AI | Up to $17 an hour | AI evaluation specialist in the Handshake AI Fellowship |
| Handshake AI | Up to $150 an hour | Software engineer in the Handshake AI Fellowship |

Volume is large. Mercor's homepage reports more than $4 million in daily payouts, and Handshake's fellowship page reports more than $500 million in payouts and more than 188,000 fellows, who must be based in the US. Scale AI's Generative AI Data Engine page quotes investor Nat Friedman on why labs pay these rates.

> "Experts are the new GPUs." (Nat Friedman, quoted on Scale AI's Generative AI Data Engine page)

| Model | How it works | Works best for | Watch for |
| --- | --- | --- | --- |
| Per hour | You pay for contributor time | Exploratory work and red teaming | Slow throughput with no cap |
| Per task | You pay per accepted demonstration, comparison or eval item | Stable, well-specified tasks | Vendors rushing tasks to hit volume |
| Per project | Fixed fee for a defined deliverable | Benchmarks and RL environments | Scope changes priced as extras |
| Managed team | A dedicated pool billed monthly | Long programs with changing guidelines | Paying for idle capacity |

Pay only for accepted tasks where you can, and define acceptance before work starts. Our guide to [AI training data pricing models](https://sellmydatatoai.net/guides/ai-training-data-pricing-models) compares these with licensing structures.

## How do you control quality in human-generated data?

Quality in human-generated data comes from screening raters, calibrating them against a reference and measuring agreement throughout the project. Ask vendors to show each step, then audit a random sample yourself.

The InstructGPT paper is a useful benchmark. OpenAI selected contractors using a screening test and their agreement with researchers on a labeling task, onboarded them with detailed instructions and kept a shared chat room for questions. Training labelers agreed with each other 72.6% of the time, plus or minus 1.5%, and held-out labelers 77.3%, plus or minus 1.3%.

The same paper admits a cost shortcut: most comparisons were labeled by only one contractor. Vendors are often less specific. Surge AI's case study on its RLHF work for Anthropic credits proprietary quality control technology but does not explain how raters are vetted or calibrated, so ask. Meta's Llama 2 team found that different annotation platforms and vendors produced markedly different downstream model performance, and checked quality by examining 180 examples by hand.

- Screening: a test of task skill and agreement with your reference answers before anyone works on live tasks
- Gold tasks: items with known answers mixed into the queue to score each rater continuously
- Overlap: a share of tasks labeled by two or more raters so agreement can be measured
- Calibration sessions: regular reviews where raters and your researchers discuss disagreements
- Versioned guidelines: a dated guideline document, because a change in guidance changes what labels mean
- Credential checks: proof of the qualifications claimed for expert raters
- Your own audit: a random sample reviewed by your team before each payment

## How should you source red teaming data?

Source red teaming data from raters trained on a written attack taxonomy, with content warnings and wellbeing support, because the work exposes people to harmful material. Ask for the instructions raters followed along with the attacks themselves.

Anthropic's 2022 red teaming paper by Deep Ganguli and colleagues released 38,961 red team attacks and found that RLHF models became harder to red team as they scaled. It also documented the instructions, processes and statistical methods used, which is the level of detail to ask a vendor for. Mercor's homepage lists a chemicals safety role for red teaming at $65 to $75, which shows red teaming being staffed by domain experts.

## What should an expert data contract include?

An expert data contract should give you clear ownership of the outputs, protect your prompts and guidelines, and set quality and acceptance terms. Most contributors are contractors, so ownership depends on the vendor's agreements with them.

- Written IP assignments from every contributor to the vendor, passed on to you
- No reuse of your prompts, guidelines or outputs for other clients
- Exclusivity of the delivered data where you need it, especially for evaluation sets
- Rules on which models generated any outputs being rated, and that their terms allow the use
- Acceptance tests, rework at the vendor's cost and the right to audit quality records
- Contributor consent, pay and location records for regulatory and reputational checks

The model terms point matters. OpenAI's terms of use, for example, bar using output from its services to develop models that compete with OpenAI. Take independent legal advice on your own contract.

## Should you build an in-house rater pool or use a vendor?

Use a vendor when you need volume or rare expertise quickly, and build in-house when the guidelines are a competitive secret or change daily. Many labs do both.

| Factor | In-house pool | Vendor |
| --- | --- | --- |
| Speed to start | Slow: recruiting and onboarding | Fast for common domains |
| Control of guidelines | Full | Shared with the vendor |
| Rare expertise | Hard to recruit | Easier through large networks |
| Confidentiality | Highest | Depends on contract and vendor controls |
| Cost visibility | Direct pay plus management time | Bundled into the vendor's price |

## When is licensed company data a better fit?

Licensed company data is a better fit when the expertise you need already exists in completed work, such as closed legal matters, audit files or resolved support tickets. Each record pairs a real problem with a verified outcome, which commissioned raters would have to recreate.

Sell My Data To AI shortlists vetted sellers of that kind of data. Every seller passes our eight-point vetting standard before any buyer sees their data, and buyers pay nothing to submit a requirement or review a shortlist. [Submit a data requirement](https://sellmydatatoai.net/request-data) or [book a buyer consultation](https://sellmydatatoai.net/book-a-consultation?side=buyer).

## Frequently asked questions

### How much does RLHF data cost?

RLHF data is priced per comparison, per hour or per project, and none of Scale AI, Surge AI, Mercor, Turing, micro1, Invisible or Handshake AI publishes a buyer rate card. Published contributor pay sets a floor: Mercor shows a $90 an hour average contracted rate, and Handshake AI lists fellowship roles from up to $17 to up to $150 an hour.

### What is a good inter-annotator agreement rate for preference data?

There is no single standard, but published work gives reference points. OpenAI's InstructGPT paper reported that training labelers agreed with each other 72.6% of the time and held-out labelers 77.3%, and cited researcher-to-researcher agreement of 73% in earlier summarization work. Contested prompts lower agreement, so ask vendors to flag them.

### Who owns data created by contract raters?

Ownership of rater output depends on the contracts between the raters, the vendor and you. Ask the vendor to confirm it holds written IP assignments from every contributor and to assign the delivered data to you, or license it on terms you accept. Without that chain, a contributor could later dispute rights in the work. Take independent legal advice.

### Can I use outputs from another company's model in my human data?

Using another company's model outputs in human data depends on that model's terms. OpenAI's terms of use, for example, bar using output from its services to develop models that compete with OpenAI. Ask your vendor which models generated any outputs raters compare or edit, and check those terms before you train on the result.

### Should I use one human data vendor or several?

Several vendors reduce dependence on one pool and let you compare quality on the same tasks. Meta's Llama 2 team found that different annotation platforms and vendors produced markedly different downstream model performance, which argues for running a small paid test with two or three vendors before you commit volume to one.

## Sources

- [Generative AI Data Engine](https://scale.com/generative-ai-data-engine), Scale AI, accessed November 2026
- [Surge AI: Human intelligence for AGI](https://www.surgehq.ai/), Surge AI, accessed November 2026
- [How Anthropic uses Surge AI's RLHF platform to train its LLM assistant](https://www.surgehq.ai/blog/anthropic-surge-ai-rlhf-platform-train-llm-assistant-human-feedback), Surge AI, accessed November 2026
- [Mercor: Organizing human intelligence to power the AI economy](https://www.mercor.com/), Mercor, accessed November 2026
- [Services](https://www.turing.com/services), Turing, accessed November 2026
- [Frontier evaluations and RL environments for AI labs](https://www.micro1.ai/human-data), micro1, accessed November 2026
- [Applied AI research for enterprise](https://invisibletech.ai/), Invisible Technologies, accessed November 2026
- [Handshake Enterprise AI](https://joinhandshake.com/ai/enterprise/), Handshake, accessed November 2026
- [Handshake AI program](https://joinhandshake.com/ai/), Handshake, accessed November 2026
- [Training language models to follow instructions with human feedback](https://arxiv.org/abs/2203.02155), arXiv (Ouyang et al., OpenAI, 2022), accessed November 2026
- [Llama 2: Open foundation and fine-tuned chat models](https://arxiv.org/abs/2307.09288), arXiv (Touvron et al., Meta, 2023), accessed November 2026
- [Red teaming language models to reduce harms: methods, scaling behaviors, and lessons learned](https://arxiv.org/abs/2209.07858), arXiv (Ganguli et al., Anthropic, 2022), accessed November 2026
- [Terms of use](https://openai.com/policies/terms-of-use/), OpenAI, accessed November 2026

---

Book a free consultation: https://sellmydatatoai.net/book-a-consultation