Buy private evaluation datasets for AI models
Last updated How we research
In short
Evaluation datasets are sets of tasks with expected answers and scoring rules, kept out of training so they measure what a model can really do. Their value depends on staying private: a 2024 study by Hugh Zhang and colleagues found accuracy drops of up to 8% when leading models were tested on fresh grade school maths problems instead of a public benchmark.
Key takeaways
- AI teams use evaluation datasets for (1) measuring model capability on private tasks, (2) regression testing between model versions and (3) grading agents with automatic or rubric scoring.
- Usually priced per task or per set, with exclusive access priced higher.
- Common delivery formats: JSONL of prompts, reference answers and metadata, rubrics and scoring scripts and containerized tasks with unit tests.
- Check before you license evaluation datasets: (1) tasks never published or shared with other buyers, (2) reference answers checked by a second expert and (3) scoring rules written down and reproducible.
- Every seller we shortlist passes 8 vetting checks before you see a sample.
Evaluation datasets: buyer's spec
- Used for
- Measuring model capability on private tasks
- Regression testing between model versions
- Grading agents with automatic or rubric scoring
- Procurement tests comparing vendors' models
- Delivery formats
- JSONL of prompts, reference answers and metadata
- Rubrics and scoring scripts
- Containerized tasks with unit tests
- Inter-annotator agreement records
- Usually priced
- Per task or per set, with exclusive access priced higher
- Check before you sign
- Tasks never published or shared with other buyers
- Reference answers checked by a second expert
- Scoring rules written down and reproducible
- Exclusivity and non-disclosure terms in the license
- US copyright in each task held by the seller, as work made for hire or by signed assignment
- Difficulty spread across the set
- Author and provenance of each task
Why AI teams work with us
- 1
Every seller of evaluation datasets we shortlist has passed eight vetting checks, from authority to sign through to a re-identification test.
- 2
Each shortlist comes with a vetting file per seller: rights evidence, provenance record, de-identification method, data manifest and proposed license terms.
- 3
Samples are drawn at random across the archive and labeled with the sampling method, so you can judge the whole package rather than its best rows.
- 4
You contract directly with the seller, and we help agree acceptance tests, staged payments and remedies before any data is delivered.
What is an evaluation dataset?
An evaluation dataset is a fixed set of tasks, each with an expected answer and a rule for scoring a model's response. It is used to measure a model, not to train it, so it must be kept apart from training data as a hold-out set.
DataVendor lists evaluation datasets among the assets labs buy and states that post-training work now creates stronger demand for evals. The glossary entry on the evaluation dataset gives a short definition.
Why does an evaluation dataset need to stay private?
An evaluation dataset needs to stay private because once its tasks appear in training data, scores stop measuring ability and start measuring memory. Researchers call this contamination.
The effect can be measured. In a 2024 study, Hugh Zhang and colleagues built GSM1k to mirror the public GSM8k maths benchmark and observed accuracy drops of up to 8% for leading models, with several model families showing evidence of systematic overfitting across almost all model sizes.
What makes a good evaluation dataset?
A good evaluation dataset is hard, unambiguous and checked by experts. OpenAI's SWE-bench Verified, published in August 2024, shows the effort involved: OpenAI worked with 93 software developers experienced in Python to screen tasks and kept 500 samples verified as non-problematic.
Rubric-scored tasks need the rubric itself to be tested. Two experts scoring the same responses should reach similar marks, and the seller should report how often they agreed.
| Property | What to look for |
|---|---|
| Ambiguity | Each task has one defensible answer or a written rubric |
| Verification | Reference answers checked by a second expert |
| Difficulty | A spread of tasks, not only easy or impossible ones |
| Coverage | Tasks that match the deployment you care about |
| Freshness | No overlap with public benchmarks or web text |
How are evaluation datasets priced?
Evaluation datasets are priced per task or per set, and exclusivity weighs heavily because a shared evaluation set loses value each time it is shared. ddiy.co's 2026 pricing guide puts an exclusive deal at about 2.5 to 5 times a non-exclusive one across company data.
Our guide to exclusive vs non-exclusive data licenses sets out how the terms differ, and AI training data pricing models compares per-task and flat-fee structures.
Who builds evaluation datasets?
Evaluation datasets are built by AI labs in-house, by training-data companies that recruit domain experts, and by companies whose own records contain tasks with known answers. A resolved support ticket, a closed legal matter or a fixed software bug each pairs a problem with a verified outcome.
Company records need conversion before they work as evaluations. Each task must be rewritten to stand alone, the expected answer separated out and a scoring rule written. Ask the seller who did that work and whether the original records were also licensed to anyone for training. If outside experts wrote the tasks, check for signed copyright assignments, because the US Copyright Office's Circular 30 treats commissioned work as made for hire only in nine listed categories, and only with a signed written agreement.
How should evaluation data be delivered?
Evaluation data should be delivered with tasks, reference answers and scoring code in separate files, each task carrying a stable ID, author, difficulty label and creation date. For agent tasks, the environment and tests should come in a container that reproduces the scoring on your own systems.
What should you check before buying an evaluation dataset?
Before buying an evaluation dataset, ask who wrote each task, whether any task has been shown to another buyer and how the reference answers were checked. Search a sample of tasks against public benchmarks and the open web for overlap.
Then fix the terms: the exclusivity period, a ban on the seller reusing tasks and what happens if a leak is found. Read how to evaluate a dataset sample, or book a free consultation to source vetted evaluation sets.
Frequently asked questions
What is the difference between training data and evaluation data?
How do I know an evaluation set has not leaked?
Who writes good evaluation tasks?
Should I buy an evaluation set exclusively?
Sources
- A careful examination of large language model performance on grade school arithmetic, arXiv (Zhang et al., 2024), accessed November 2026
- Introducing SWE-bench Verified, OpenAI, accessed November 2026
- AI training data marketplaces, DataVendor, accessed November 2026
- How much is company data worth to AI? (2026 prices), ddiy.co, accessed November 2026
- Works Made for Hire (Circular 30), US Copyright Office, accessed November 2026
For AI teams
Ready to source?
Book a consultation to brief us. We shortlist vetted sellers, each with a vetting file covering rights evidence, provenance and a representative sample.
Related guides
- How to evaluate a dataset sample before you buy
Buying data
- Exclusive vs non-exclusive data licenses
Selling data
- AI training data pricing models
Pricing and value
- Agent trajectories and RL environments explained
Data types
- How to buy AI training data
Buying data