# LLM training data

URL: https://aidataforsale.com/llm-training-data
Publisher: AI Data For Sale (Marketing Signals Ltd)
Updated: 2026-11-10

The data large language models need at each stage, from pre-training text to fine-tuning, preference, evaluation and agent data, and where licensed data fits.

What data a large language model needs from pre-training to agents, and where licensed private data makes the difference.

## In short

- LLMs use different data at each stage: large text corpora for pre-training, examples for fine-tuning, comparisons for preference training, and held-out sets for evaluation.
- Epoch AI estimated the stock of public human-written text at about 300 trillion tokens, likely to be fully used between 2026 and 2032.
- As public text runs short, private, specialist and multimodal data matters more.
- Agent training needs records of real work: tool use, decisions and outcomes.

## Data by training stage

| Stage | What the data looks like | Licensable sources |
| --- | --- | --- |
| [Pre-training](https://aidataforsale.com/glossary/pre-training) | Very large volumes of text, code and other media | [Books and long-form text](https://aidataforsale.com/data/books-and-long-form-text), [multilingual text](https://aidataforsale.com/data/multilingual-and-low-resource-language-data), [source code](https://aidataforsale.com/data/source-code-datasets) |
| [Supervised fine-tuning](https://aidataforsale.com/glossary/supervised-fine-tuning) | Prompt and response examples of the target behavior | [Conversational data](https://aidataforsale.com/data/conversational-data), [customer support tickets](https://aidataforsale.com/data/customer-support-tickets), [expert-annotated data](https://aidataforsale.com/data/expert-annotated-data) |
| [Preference training](https://aidataforsale.com/glossary/rlhf) | Comparisons and ratings of model outputs | [RLHF and preference data](https://aidataforsale.com/data/rlhf-preference-data) |
| [Evaluation](https://aidataforsale.com/glossary/evaluation-dataset) | Held-out tasks with trusted answers | [Evaluation datasets](https://aidataforsale.com/data/evaluation-datasets) |
| Agents and RL | Records of multi-step work with tools and outcomes | [Agent trajectories](https://aidataforsale.com/data/agent-trajectory-data), [software issue history](https://aidataforsale.com/data/software-issue-history), [workplace chat](https://aidataforsale.com/data/workplace-chat-data) |

## Why public text is not enough

Epoch AI estimated the effective stock of human-generated public text at about 300 trillion tokens, and projected it would be fully used at some point between 2026 and 2032 ([Epoch AI](https://epoch.ai/blog/will-we-run-out-of-data-limits-of-llm-scaling-based-on-human-generated-data)). At the same time, websites are restricting AI crawlers: one study found that in a single year 5% or more of all tokens in the C4 dataset, and 28% or more of its most critical sources, became fully restricted ([Consent in Crisis](https://arxiv.org/abs/2407.14933)).

Synthetic data helps, but models trained recursively on generated data can degrade ([Nature](https://www.nature.com/articles/s41586-024-07566-y)). Real human data, especially private and specialist data, fills the gap. See [synthetic vs licensed training data](https://aidataforsale.com/learn/synthetic-vs-licensed-training-data).

## What makes LLM data valuable

- **Unseen:** not already on the public web
- **Specialist:** domain, language or task coverage others lack
- **Structured:** turns, roles, outcomes and metadata
- **Clean rights:** documented provenance and license

## How we help

Tell us the stage, domain and volume you need. We find holders, check their rights and privacy handling, and arrange samples. [Request data](https://aidataforsale.com/request-data).

## Read next

- [AI training data (glossary)](https://aidataforsale.com/glossary/ai-training-data)
- [What is AI training data?](https://aidataforsale.com/learn/what-is-ai-training-data)
- [Datasets for sale](https://aidataforsale.com/datasets-for-sale)

## Questions

### How much data does it take to train an LLM?

Pre-training uses trillions of tokens; fine-tuning can use thousands to millions of examples. The right amount depends on the model and the goal.

### Is licensed data better than scraped data?

It has clear rights and can include material that was never public. Court rulings, including Bartz v. Anthropic, have penalized pirated sources. See the [lawsuit tracker](https://aidataforsale.com/ai-copyright-lawsuits).

## Sources

- [Epoch AI: Will we run out of data? Limits of LLM scaling based on human-generated data (June 2024)](https://epoch.ai/blog/will-we-run-out-of-data-limits-of-llm-scaling-based-on-human-generated-data)
- [Longpre et al.: Consent in Crisis, the rapid decline of the AI data commons (Data Provenance Initiative, July 2024)](https://arxiv.org/abs/2407.14933)
- [Shumailov et al.: AI models collapse when trained on recursively generated data (Nature, 2024)](https://www.nature.com/articles/s41586-024-07566-y)
- [Ouyang et al.: Training language models to follow instructions with human feedback (2022)](https://arxiv.org/abs/2203.02155)