The Data Layer, Mapped: From a Web Crawl to a Training Batch
A frontier model is made of compute, code and data, and data is the part labs say least about. Before a single GPU starts training, petabytes of web pages, books, code, licensed archives, synthetic text and human-written examples have to be found, cleaned, deduplicated, mixed and turned into tokens. This first part of a new series maps that supply chain end to end, shows how little of the raw web survives it, and follows the money and the lawsuits that now surround it.
1. Case file
2. Timeline
Training data, 2008 โ 2026
Tap a milestone. Colour: open datasets, industry deals, law and limits.
Each generation of open datasets has been bigger and, above all, better filtered. C4 (2019) cleaned one Common Crawl snapshot with simple rules; The Pile (2020) mixed 22 curated sources; RefinedWeb (2023) showed that well-filtered web data alone could match curated mixtures; FineWeb, Dolma and DCLM (2024) published their whole pipelines and pushed quality with model-based filters. Meanwhile frontier labs went quiet about their own data, licensing deals multiplied, and courts began to rule on what training is allowed to use.
3. The supply chain
Follow one web page into a training batch
Eight stages. Step through to see what happens to a page, and what gets thrown away. Simplified.
4. How much survives
Most of the web is not fit for training: navigation menus, spam, machine-translated filler, near-identical copies, adult content, personal data, boilerplate. Every stage of the pipeline removes some of it. NVIDIA's Nemotron-CC paper notes that recent datasets such as DCLM and FineWeb-Edu won benchmark gains through aggressive model-based filtering "at the cost of removing 90% of data", which is exactly the trade-off a long training run cannot afford: too strict and you run out of tokens; too loose and the model learns from junk.
The data funnel
An illustration of the share of raw crawled text left after each stage. Move the slider from lenient to strict quality filtering. Ratios are our own, in the range dataset papers describe.
5. Three kinds of data
What goes in at each stage
An illustrative mix by training stage, not any lab's real recipe. Most labs do not publish theirs.
6. The open datasets
Open pre-training corpora
Sizes as published. Different units (tokens vs bytes) and tokenisers make them only roughly comparable. Tap a bar.
7. How big is 15 trillion tokens?
Token converter
Our arithmetic with round assumptions: ~0.75 English words per token, ~100,000 words per book, ~5 billion words in English Wikipedia, ~1,000 words per web page, 4 bytes of text per token.
8. The data economy
Data used to be scraped for free. It is now also bought. Labs pay publishers and platforms for licensed access, and pay a fast-growing industry of data companies for human-written and human-judged examples, increasingly from experts such as doctors, lawyers and mathematicians.
Deals and data companies
Reported figures. Hatched bars are press reports or third-party estimates, not company filings.
9. Who discloses what
What each organisation has published about each stage. Tap a cell. โ detailed ยท โ partial ยท โ little or nothing.
The pattern is clear: the most open data comes from organisations that do not sell frontier models (AI2, Hugging Face, NVIDIA, academic groups), while the leading labs describe their data in a paragraph. Meta's Llama 3 paper is the most detailed frontier disclosure, with its total token count, filtering approach and a rough mix (about half general knowledge, a quarter maths and reasoning, 17% code, 8% multilingual), but not its sources.
10. Law and limits
11. How the data layer feeds the ML layer
Two layers, many connections
Tap a link to see where data meets the ML stack.
12. The series
Inside the Data Layer: eight parts
13. Reading the numbers
- Primary
- FineWeb ~15T tokens, 96 snapshots, ~44 TB; FineWeb-Edu 1.3T (paper, NeurIPS 2024). Nemotron-CC 6.3T incl. 1.9T synthetic; model-based filtering in DCLM/FineWeb-Edu removes ~90% (paper). Dolma ~3T tokens, 5B documents, 7 sources (AI2). Epoch AI: ~300T tokens (100Tโ1,000T), used up 2026โ2032. Llama 3: ~15T tokens and mix shares (Meta paper). Reddit IPO filing: $203M aggregate data-licensing contracts over 2โ3 years.
- Reported โ ๏ธ
- MetaโScale ~$14.3B for 49%, ~$29B valuation, Scale 2024 revenue ~$870M (Reuters and others); customers pulling back (reports). RedditโGoogle ~$60M/year (Reuters, unconfirmed); News CorpโOpenAI >$250M over five years (WSJ). Surge AI 2024 revenue >$1B (press; Sacra estimates ~$1.2B). Mercor ~$2B gross run rate, before contractor pay (press). Anthropic settlement $1.5B, ~$3,000 ร ~500K books (AP and others).
- From the papers, not re-checked โ ๏ธ
- C4 ~750 GB (the Dolma paper lists 790 GB uncompressed); The Pile 825 GiB, 22 sources; RefinedWeb ~5T tokens (600B released); DCLM pool ~240T tokens, DCLM-Baseline ~3.8T.
- Our illustrations
- The supply-chain example, funnel ratios, stage mixes and token conversions.
- Datasets
- C4 (T5 paper) ยท The Pile ยท RefinedWeb
- Dolma ยท FineWeb ยท DCLM ยท Nemotron-CC
- The Llama 3 herd of models
- Limits, deals and law
- Epoch AI: Will we run out of data?
- TechCrunch: Meta and Scale AI
- CJR: Reddit's AI licensing deals
- Quartz: AI training-data licensing market
- Silicon: Surge AI funding
- AP via TPR: Anthropic to pay authors $1.5B
- BetaNews: settlement receives final approval
- Inside the ML Layer at a Glance ยท Producing Intelligence Part 2: data