Evaluation dataset
By Gareth HoyleLast updated How we research
An evaluation dataset is a set of tasks with expected answers and scoring rules, used to measure how well a model performs rather than to train it.
An evaluation dataset is a collection of tasks, each with an expected answer or a scoring rule, used to measure a model's performance. It is kept separate from training data so that scores reflect real ability, not memory.
Why are evaluation datasets valuable?
Evaluation datasets are valuable because public benchmarks leak into web-scale training data over time, so labs need fresh, private tests in specialist domains. A set built from real expert work, with answers checked by qualified reviewers, shows whether a model can do the job.
What does a good evaluation dataset contain?
- Tasks drawn from real work, not invented examples
- Reference answers or rubrics written by experts
- Scoring rules that two reviewers would apply the same way
- Metadata on difficulty and domain
- Confirmation that the set has never been published
Sellers should license evaluation sets on terms that bar training on them, and buyers can submit a data requirement for a specific domain. See hold-out set and evaluation datasets.