# Pre-training

URL: https://sellmydatatoai.net/glossary/pre-training

Pre-training is the first, largest stage of building a language model, in which it learns from trillions of tokens of general text and code.

Pre-training is the first stage of building a foundation model, in which the model learns general patterns of language, code or images by predicting missing or next tokens across a very large corpus. It uses far more data and compute than any later stage. Meta said in April 2024 that [Llama 3 was pretrained on over 15 trillion tokens](https://ai.meta.com/blog/meta-llama-3/) collected from publicly available sources.

## What data do buyers want for pre-training?

Buyers want large volumes of clean, varied, rights-cleared text and code for pre-training, such as book archives, publishing back catalogs, code repositories and long-form documents.

## How is pre-training different from fine-tuning?

Pre-training builds broad capability. [Fine-tuning](https://sellmydatatoai.net/glossary/fine-tuning), including [supervised fine-tuning](https://sellmydatatoai.net/glossary/supervised-fine-tuning) and [RLHF](https://sellmydatatoai.net/glossary/rlhf), adapts the model afterward using much smaller, higher-quality sets, where company records and expert data earn more per item. Our guide to [AI training data pricing models](https://sellmydatatoai.net/guides/ai-training-data-pricing-models) compares how each stage is paid for.