Benchmark contamination
By Gareth HoyleLast updated How we research
Benchmark contamination is when test questions or close copies appear in a model's training data, so its score reflects memory rather than ability.
Benchmark contamination is the presence of an evaluation set's items, or close paraphrases of them, in a model's training data. A contaminated model can score well by recalling answers it has seen, so the benchmark overstates its real ability.
How large is the effect?
The effect can be measured. In a 2024 study, Hugh Zhang and colleagues built GSM1k to mirror the public GSM8k math benchmark and observed accuracy drops of up to 8% for leading models, with several model families showing evidence of systematic overfitting.
Why does contamination matter for evaluation data sales?
Contamination matters because an evaluation dataset is worth most while it stays private. Once its tasks leak onto the web or into another buyer's training data, it stops measuring anything. Sellers of evaluation sets therefore keep records of who has seen each item, offer exclusive licenses and keep a hold-out set back.
Buyers should search a sample of tasks against public benchmarks and web text before paying. Our guide to buying evaluation datasets covers the checks.