- LLMs use different data at each stage: large text corpora for pre-training, examples for fine-tuning, comparisons for preference training, and held-out sets for evaluation.
- Epoch AI estimated the stock of public human-written text at about 300 trillion tokens, likely to be fully used between 2026 and 2032.
- As public text runs short, private, specialist and multimodal data matters more.
- Agent training needs records of real work: tool use, decisions and outcomes.
Data by training stage
| Stage | What the data looks like | Licensable sources |
|---|---|---|
| Pre-training | Very large volumes of text, code and other media | Books and long-form text, multilingual text, source code |
| Supervised fine-tuning | Prompt and response examples of the target behavior | Conversational data, customer support tickets, expert-annotated data |
| Preference training | Comparisons and ratings of model outputs | RLHF and preference data |
| Evaluation | Held-out tasks with trusted answers | Evaluation datasets |
| Agents and RL | Records of multi-step work with tools and outcomes | Agent trajectories, software issue history, workplace chat |
Why public text is not enough
Epoch AI estimated the effective stock of human-generated public text at about 300 trillion tokens, and projected it would be fully used at some point between 2026 and 2032 (Epoch AI). At the same time, websites are restricting AI crawlers: one study found that in a single year 5% or more of all tokens in the C4 dataset, and 28% or more of its most critical sources, became fully restricted (Consent in Crisis).
Synthetic data helps, but models trained recursively on generated data can degrade (Nature). Real human data, especially private and specialist data, fills the gap. See synthetic vs licensed training data.
What makes LLM data valuable
- Unseen: not already on the public web
- Specialist: domain, language or task coverage others lack
- Structured: turns, roles, outcomes and metadata
- Clean rights: documented provenance and license
How we help
Tell us the stage, domain and volume you need. We find holders, check their rights and privacy handling, and arrange samples. Request data.
Read next
Questions
How much data does it take to train an LLM?
Pre-training uses trillions of tokens; fine-tuning can use thousands to millions of examples. The right amount depends on the model and the goal.
Is licensed data better than scraped data?
It has clear rights and can include material that was never public. Court rulings, including Bartz v. Anthropic, have penalized pirated sources. See the lawsuit tracker.
Sources
- Epoch AI: Will we run out of data? Limits of LLM scaling based on human-generated data (June 2024)
- Longpre et al.: Consent in Crisis, the rapid decline of the AI data commons (Data Provenance Initiative, July 2024)
- Shumailov et al.: AI models collapse when trained on recursively generated data (Nature, 2024)
- Ouyang et al.: Training language models to follow instructions with human feedback (2022)