How to design a training data pilot before a full license

A pilot is the cheapest point at which to find out that a dataset does not help your model. After a full license, the money is spent. After training, the data is in the weights.

This guide explains how to size a pilot, set up the evaluation, write acceptance criteria and structure payment. It builds on our guide to evaluating a dataset sample, which covers checking the records themselves.

What is a training data pilot?

A training data pilot is a limited, paid test of a dataset on an experiment both sides agree in advance. It sits between a sample review and a full license.

Nixon Peabody's October 2026 diligence alert describes the same staging from the legal side: the buyer starts with a limited sample in a controlled environment, validates rights and quality, and expands access only as agreed conditions are met.

How big should a training data pilot be?

A pilot needs two samples of different sizes: an audit sample to estimate defect rates and a training slice to measure model effect. Size the audit sample with simple statistics and the training slice by the training use.

Sizing the audit sample

The audit sample is a random draw of records that your team reviews by hand or by script for errors, personal data, duplicates and off-spec records. Its size sets how precisely you can estimate a defect rate.

The figures follow from the standard formula for a proportion, 1.96 times the square root of p times 1 minus p over n. Margins shrink when the true rate is far from 50%, so they are an upper bound.

For rare defects such as residual personal data, use the rule of three. As Thomas Newman summarized in JAMA in 1995, citing Hanley and Lippman-Hand, if no events are observed in n trials, the upper limit of the 95% confidence interval for their frequency is about 3 divided by n. An audit of 300 random records with no personal data found supports a rate below about 1%, and 3,000 clean records supports below about 0.1%.

Sizing the training slice

The training slice should be big enough to move your evaluation if the data is useful. For fine-tuning data, published work suggests that can be small: the LIMA paper fine-tuned a 65 billion parameter model on 1,000 curated examples, and Meta's Llama 2 team stopped collecting supervised fine-tuning data at 27,540 annotations, finding that tens of thousands were enough for a high-quality result.

Pre-training data is harder to pilot, because small slices rarely move a large model. A common approach is to train a smaller proxy model on mixtures with and without the candidate data. Microsoft's Phi-4 report describes choosing its data mixture through ablations at 7 billion parameters on a shorter 1 trillion token run, after observing a high rank correlation between 7 billion and 14 billion parameter results across mixtures.

How do you set up a held-out evaluation for a pilot?

A held-out evaluation is a fixed test set that neither the seller nor the pilot data has touched. Record your current model's score on it before the pilot so you can measure the change.

1. 1

   Fix the evaluation set

   Choose or build tasks that reflect the deployment you care about and freeze them before you see the pilot data.

2. 2

   Record a baseline

   Score your current model, and record the settings, seed and compute used.

3. 3

   Run a control

   Train on the same compute budget with data you already hold, so you separate the effect of new data from the effect of extra training.

4. 4

   Run the pilot

   Train with the pilot slice under the same settings, and repeat with more than one seed if budget allows.

5. 5

   Check for contamination

   Search the pilot slice for overlap with the evaluation set, including paraphrases, before you trust any gain.

Exact matching is not enough. Shuo Yang and colleagues showed in 2023 that paraphrased or translated test items slip past n-gram decontamination, and found that 8% to 18% of the HumanEval coding benchmark overlapped with the RedPajama-Data-1T and StarCoder-Data pre-training sets. Our guide to buying evaluation datasets covers contamination in more depth.

Acceptance criteria should be written before the pilot starts and should cover both the data and the model effect. The thresholds below are examples to adapt, not market standards.

Attach the same criteria to the full license as its acceptance schedule. A pilot that passes on criteria you never wrote into the contract gives you no remedy if the full delivery differs from the slice.

How should payment be staged around a pilot?

Payment around a pilot is usually staged in three parts: a pilot fee, a license fee on conversion and a holdback released when the full delivery passes acceptance. Nixon Peabody lists holdbacks among the tools for allocating risk, alongside indemnities, liability caps, insurance and termination rights.

Whether the pilot fee is credited, and how large the holdback is, are commercial points to negotiate. Our guide to AI training data pricing models explains how flat fees, unit prices and recurring access interact with staged payment.

What rights does a pilot license need?

A pilot license needs testing rights only, with clear rules for what happens to anything trained on the slice if you do not convert. Sellers will ask for this, and it protects you from a dispute later.

• Use limited to the agreed experiment on named models
• Deletion of the slice and of pilot-trained weights if you do not convert, with a written certificate
• No onward transfer outside the approved environment
• Confidentiality of results, or aggregate results only if the seller prefers
• A conversion window during which the pilot price and terms hold

In the US, the license rather than fair use is what governs your pilot. Our AI training data lawsuits tracker shows why buyers prefer contract certainty while the case law develops.

What are the common mistakes in pilot design?

• Letting the seller choose the pilot slice instead of drawing it at random
• Measuring against no baseline or control, so extra training gets credited to the data
• Writing acceptance criteria after seeing the results
• Using an evaluation set the seller has seen or that overlaps public benchmarks
• Signing a full license whose acceptance schedule does not match the pilot criteria

How does Sell My Data To AI support pilots?

Sell My Data To AI shortlists vetted sellers and helps both sides prepare for a pilot. Every seller passes our eight-point vetting standard, which includes a representative sample drawn at random across the archive and a check that the manifest matches what is delivered. Buyers see redacted samples under a mutual NDA, and you sign any pilot or full license directly with the seller.

To scope a pilot, book a buyer consultation or submit a data requirement. Our procurement guide shows where the pilot fits in the wider process.