Pre-training
By Gareth HoyleLast updated How we research
Pre-training is the first, largest stage of building a language model, in which it learns from trillions of tokens of general text and code.
Pre-training is the first stage of building a foundation model, in which the model learns general patterns of language, code or images by predicting missing or next tokens across a very large corpus. It uses far more data and compute than any later stage. Meta said in April 2024 that Llama 3 was pretrained on over 15 trillion tokens collected from publicly available sources.
What data do buyers want for pre-training?
Buyers want large volumes of clean, varied, rights-cleared text and code for pre-training, such as book archives, publishing back catalogs, code repositories and long-form documents.
How is pre-training different from fine-tuning?
Pre-training builds broad capability. Fine-tuning, including supervised fine-tuning and RLHF, adapts the model afterward using much smaller, higher-quality sets, where company records and expert data earn more per item. Our guide to AI training data pricing models compares how each stage is paid for.