Hold-out set
By Gareth HoyleLast updated How we research
A hold-out set is a portion of data kept out of training so it can test how well a model performs on examples it has never seen.
A hold-out set is a portion of data withheld from training and used only to test a model, so its results show how well the model generalizes to unseen examples. Data is commonly split into training, validation and test sets: the validation set guides tuning and the test set gives the final score.
What is contamination?
Contamination is what happens when hold-out examples, or near copies of them, end up in training data. Scores then overstate real ability. It is a growing problem for large language models trained on web-scale text, because published benchmarks are copied across the internet.
Why does a hold-out set matter in a data deal?
A hold-out set matters in a data deal in two ways. Buyers can hold back part of a delivered dataset to test whether it improves their models before final payment. Sellers of evaluation datasets need contract terms that keep the set out of training.
See how to evaluate a dataset sample and representative sample.