Home โ€บ Blog โ€บ The data layer, mapped
Inside the Data Layer ยท Part 1 ยท The map

The Data Layer, Mapped: From a Web Crawl to a Training Batch

A frontier model is made of compute, code and data, and data is the part labs say least about. Before a single GPU starts training, petabytes of web pages, books, code, licensed archives, synthetic text and human-written examples have to be found, cleaned, deduplicated, mixed and turned into tokens. This first part of a new series maps that supply chain end to end, shows how little of the raw web survives it, and follows the money and the lawsuits that now surround it.

FL
FrontierAGI Team
โš ๏ธ How this was researched (as of 27 Oct 2026) Sources are the dataset papers (C4, The Pile, RefinedWeb, Dolma, FineWeb, DCLM, Nemotron-CC), Meta's Llama 3 paper, Epoch AI's data-stock study, and Reuters, AP and other reporting on the Scale AI deal, licensing agreements and the authors' settlement. Frontier labs disclose little about their data; where we infer, we say so. Revenue figures for private data companies are press reports or estimates (โš ๏ธ). Funnel ratios and data mixes are our own illustrations. Disclosure: Claude, made by Anthropic, helps write this site; this article reports Anthropic's copyright settlement from news coverage.

1. Case file

๐Ÿ—ƒ๏ธ
The data layer, October 2026The inputs to every layer of the ML stack
Llama 3 pre-training data~15T tokens; exact composition undisclosed
FineWeb (Hugging Face, 2024)~15T tokens from 96 Common Crawl snapshots
Nemotron-CC (NVIDIA, Dec 2024)6.3T tokens, incl. 1.9T synthetic
Meta โ†’ Scale AI (Jun 2025)~$14.3B for 49%, ~$29B valuation
Anthropic authors' settlement (Sep 2025)$1.5B, ~$3,000 per book, ~500K books
Public human text (Epoch AI, 2024)~300T tokens; fully used 2026โ€“2032 if trends hold

2. Timeline

Training data, 2008 โ†’ 2026

Tap a milestone. Colour: open datasets, industry deals, law and limits.

Tap a milestone.

Each generation of open datasets has been bigger and, above all, better filtered. C4 (2019) cleaned one Common Crawl snapshot with simple rules; The Pile (2020) mixed 22 curated sources; RefinedWeb (2023) showed that well-filtered web data alone could match curated mixtures; FineWeb, Dolma and DCLM (2024) published their whole pipelines and pushed quality with model-based filters. Meanwhile frontier labs went quiet about their own data, licensing deals multiplied, and courts began to rule on what training is allowed to use.

3. The supply chain

Follow one web page into a training batch

Eight stages. Step through to see what happens to a page, and what gets thrown away. Simplified.

4. How much survives

Most of the web is not fit for training: navigation menus, spam, machine-translated filler, near-identical copies, adult content, personal data, boilerplate. Every stage of the pipeline removes some of it. NVIDIA's Nemotron-CC paper notes that recent datasets such as DCLM and FineWeb-Edu won benchmark gains through aggressive model-based filtering "at the cost of removing 90% of data", which is exactly the trade-off a long training run cannot afford: too strict and you run out of tokens; too loose and the model learns from junk.

The data funnel

An illustration of the share of raw crawled text left after each stage. Move the slider from lenient to strict quality filtering. Ratios are our own, in the range dataset papers describe.

5. Three kinds of data

๐ŸŒ Natural dataText and code that people wrote for their own reasons: the web, books, papers, code repositories, forums, licensed archives. The bulk of pre-training.Part 2 (sourcing) and Part 3 (filtering)
๐Ÿค– Synthetic dataText generated by models: rephrased web pages, textbooks written to order, reasoning traces, code with tests, answers checked by verifiers. Growing fast in every stage.Part 4
๐Ÿ‘ฉโ€๐Ÿซ Human dataWritten or judged by paid people for training: demonstrations, preference rankings, expert answers, RL task design. Small in tokens, large in cost and influence.Part 5

What goes in at each stage

An illustrative mix by training stage, not any lab's real recipe. Most labs do not publish theirs.

6. The open datasets

Open pre-training corpora

Sizes as published. Different units (tokens vs bytes) and tokenisers make them only roughly comparable. Tap a bar.

Tap a dataset.

7. How big is 15 trillion tokens?

Token converter

Our arithmetic with round assumptions: ~0.75 English words per token, ~100,000 words per book, ~5 billion words in English Wikipedia, ~1,000 words per web page, 4 bytes of text per token.

8. The data economy

Data used to be scraped for free. It is now also bought. Labs pay publishers and platforms for licensed access, and pay a fast-growing industry of data companies for human-written and human-judged examples, increasingly from experts such as doctors, lawyers and mathematicians.

Deals and data companies

Reported figures. Hatched bars are press reports or third-party estimates, not company filings.

๐Ÿ’ก The Scale AI shockIn June 2025 Meta paid about $14.3 billion for 49% of Scale AI, valuing it at about $29 billion, and Scale's chief executive Alexandr Wang left to help lead Meta's AI work. Reports said Google and OpenAI, both Scale customers, moved to wind down work with it over concerns about revealing their research priorities to a competitor โš ๏ธ, and rivals such as Surge AI and Mercor grew quickly. The lesson for the data layer: who labels your data can see what you are trying to teach.

9. Who discloses what

What each organisation has published about each stage. Tap a cell. โ— detailed ยท โ— partial ยท โ€” little or nothing.

Tap a cell.

The pattern is clear: the most open data comes from organisations that do not sell frontier models (AI2, Hugging Face, NVIDIA, academic groups), while the leading labs describe their data in a paragraph. Meta's Llama 3 paper is the most detailed frontier disclosure, with its total token count, filtering approach and a rough mix (about half general knowledge, a quarter maths and reasoning, 17% code, 8% multilingual), but not its sources.

10. Law and limits

CopyrightIn June 2025 a US judge ruled in a case against Anthropic that training on books was fair use, but that downloading millions of them from pirate sites was not. Anthropic then agreed a $1.5 billion class settlement, about $3,000 for each of roughly 500,000 books, and to destroy the downloaded files; it later received final approval, with some authors opting out to sue separately. Many other cases against AI companies remain open. Part 8 covers the legal map.
The data wallEpoch AI estimated in 2024 that the effective stock of public human text is about 300 trillion tokens (90% range 100Tโ€“1,000T), and that at past trends models would use all of it between 2026 and 2032, sooner if heavily over-trained. Labs' responses are the rest of this series: better filtering, repeating good data, licensed and private sources, synthetic data and human experts.

11. How the data layer feeds the ML layer

Two layers, many connections

Tap a link to see where data meets the ML stack.

12. The series

Inside the Data Layer: eight parts

VerdictCompute is bought and code can be copied, but data is where frontier labs most clearly differ and say least. The open datasets show the craft: most of the web is thrown away, and what is kept is filtered, deduplicated and mixed with care. The frontier has moved on to sources money and models can create: licensed archives, expert human data and synthetic text. With public human text approaching its limits and courts drawing lines around how it is obtained, the data layer is becoming the slowest, most expensive and most contested part of the AI factory.

13. Reading the numbers

Primary
FineWeb ~15T tokens, 96 snapshots, ~44 TB; FineWeb-Edu 1.3T (paper, NeurIPS 2024). Nemotron-CC 6.3T incl. 1.9T synthetic; model-based filtering in DCLM/FineWeb-Edu removes ~90% (paper). Dolma ~3T tokens, 5B documents, 7 sources (AI2). Epoch AI: ~300T tokens (100Tโ€“1,000T), used up 2026โ€“2032. Llama 3: ~15T tokens and mix shares (Meta paper). Reddit IPO filing: $203M aggregate data-licensing contracts over 2โ€“3 years.
Reported โš ๏ธ
Metaโ€“Scale ~$14.3B for 49%, ~$29B valuation, Scale 2024 revenue ~$870M (Reuters and others); customers pulling back (reports). Redditโ€“Google ~$60M/year (Reuters, unconfirmed); News Corpโ€“OpenAI >$250M over five years (WSJ). Surge AI 2024 revenue >$1B (press; Sacra estimates ~$1.2B). Mercor ~$2B gross run rate, before contractor pay (press). Anthropic settlement $1.5B, ~$3,000 ร— ~500K books (AP and others).
From the papers, not re-checked โš ๏ธ
C4 ~750 GB (the Dolma paper lists 790 GB uncompressed); The Pile 825 GiB, 22 sources; RefinedWeb ~5T tokens (600B released); DCLM pool ~240T tokens, DCLM-Baseline ~3.8T.
Our illustrations
The supply-chain example, funnel ratios, stage mixes and token conversions.