Where the Tokens Come From: Crawlers, Licences and the Closing Web
Every training run starts with a shopping list: web pages, code, books, papers, forums, news archives, transcripts. Some of it is crawled for free, some is bought, some is public domain, and a growing share is now blocked. This part explains how crawlers collect the web, what each source is worth, how websites are fighting back, and why labs increasingly pay for data they used to take.
1. Case file
2. Timeline
Sourcing training data, 1994 โ 2026
Tap a milestone. Colour: crawling, open datasets, restrictions, deals.
3. The sourcing menu
No single source is enough. Labs blend many, each with a different mix of size, quality, legal risk and cost. The open web provides volume; code, books and papers provide reasoning and depth; forums provide conversation; licensed archives provide freshness, quality and legal cover.
Source ร trade-off
Our editorial ratings (1 = low, 5 = high). Tap a row for details.
4. How a crawler works
A crawler keeps a frontier, a queue of URLs to visit, seeded with known sites. For each URL it checks the site's robots.txt, waits a polite interval so it does not overload the server, fetches the page, stores the raw response (Common Crawl uses the WARC archive format), extracts new links and adds them to the frontier. At web scale this means billions of pages, deduplicated URLs, revisit schedules for pages that change, and sometimes a headless browser to render JavaScript.
Crawler simulator
A toy crawler over a small web. Choose which user-agent it identifies as, then step. Sites and rules are invented.
5. Common Crawl: the commons everyone uses
Common Crawl, a non-profit founded in 2008, publishes a fresh crawl roughly every month and has archived more than 10 petabytes. Most open pre-training datasets (C4, RefinedWeb, FineWeb, DCLM, Nemotron-CC) are built from it, and many labs use it alongside their own crawlers. Its September 2025 crawl held 2.39 billion pages and 421 TiB of uncompressed content. Crawls have been getting smaller, which one analysis attributes partly to more sites blocking its crawler, CCBot โ ๏ธ.
Pages per crawl
Average pages per monthly crawl. 2024โ2026 averages from a third-party analysis of Common Crawl announcements (โ ๏ธ); Sep 2025 from Common Crawl.
6. robots.txt: the web's polite "no"
Since 1994, sites have used a plain-text file, robots.txt, to tell crawlers what they may fetch; it was finally standardised as RFC 9309 in 2022. It is a request, not a lock: well-behaved crawlers obey it, others ignore it. AI companies publish separate user-agents so sites can opt out of training without disappearing from search: OpenAI's GPTBot, Anthropic's ClaudeBot, Common Crawl's CCBot, and tokens such as Google-Extended and Applebot-Extended that control AI use of content without blocking the search crawler itself.
robots.txt explorer
Edit the rules or pick a preset, then see what each crawler may fetch. Simplified matching (longest prefix wins; Allow beats Disallow on ties).
Google-Extended is a control token: Google's search crawler still fetches pages, but blocking it opts content out of Gemini training and grounding. Real crawlers also support wildcards; this matcher does not.
7. The closing web
The Data Provenance Initiative's "Consent in Crisis" study (NeurIPS 2024) audited about 14,000 web domains and found that in a single year (2023โ2024) robots.txt rules had fully restricted 5%+ of all tokens in C4, and 28%+ of its most actively maintained, critical sources; terms of service restricted 45% of C4. Then, on 1 July 2025, Cloudflare, which sits in front of roughly a fifth of websites, began blocking known AI crawlers by default for new customers and launched "pay per crawl", letting sites charge for access.
Consent in Crisis: C4 restrictions
From the paper. Note the units: tokens vs sources.
Crawl-to-referral ratios โ ๏ธ
Pages crawled for each visitor sent back, from Cloudflare Radar as reported by the press. Log scale. Figures vary widely by month and by report; Search Engine Journal found seven different published ratios for one company. Treat as orders of magnitude.
8. Buying instead of crawling
As the free web closes, labs buy access. Licences offer more than tokens: guaranteed freshness, high-quality archives behind paywalls, structured data, and legal cover. Reddit's 2024 IPO filing disclosed $203 million of data-licensing contracts over two to three years; News Corp's deal with OpenAI was reported at more than $250 million over five years. Part 1 lists more.
What does a licensed token cost?
Our own arithmetic. Pick a reported deal value and estimate how many usable tokens it buys; the price per million tokens is very sensitive to the guess.
9. Code, books and science up close
10. Languages
The web is mostly English and a handful of large languages, and crawls inherit that skew; under half of Common Crawl's pages are estimated to be English โ ๏ธ, but most other languages are thin and lower quality after filtering. Labs that care about other languages turn to national libraries, government corpora, news archives, translation and synthetic data. Governments increasingly fund this: India's language programmes (such as Bhashini and AI Kosh โ ๏ธ), Europe's multilingual projects and others aim to give their languages enough clean text to train on. Part 6 covers how language shares are set in the mix.
11. Who uses what
| Organisation | Own crawler | Sources disclosed | Licensing |
|---|---|---|---|
| OpenAI | GPTBot (documented) | Little | Many publisher deals (News Corp, AP and others) reported |
| Anthropic | ClaudeBot (documented) | Little; respects robots.txt per its docs | Book settlement; deals not widely reported โ ๏ธ |
| Googlebot + Google-Extended control | Little | Reddit deal reported | |
| Meta | Own crawler โ ๏ธ | Llama 3: public sources, ~15T tokens | Some deals reported โ ๏ธ |
| AI2, Hugging Face, NVIDIA | Mostly Common Crawl | Full pipelines published | Open data |
| BigCode, Pleias, Harvard | Software Heritage, archives | Fully documented | Open licences, opt-outs |
12. Choosing sources
What should your mix lean on?
An editorial guide.
Inside the Data Layer
13. Reading the numbers
- Primary
- Common Crawl Sep 2025: 2.39B pages, 421 TiB; archive >10 PB (Common Crawl). Consent in Crisis: ~14,000 domains; 5%+ of C4 tokens and 28%+ of critical sources fully restricted by robots.txt; 45% of C4 restricted by terms of service (NeurIPS 2024). The Stack v2: 67.5 TB raw, 32.1 TB deduplicated, ~900B tokens (BigCode/Hugging Face). Institutional Books: 983,004 volumes, 242B tokens (arXiv 2506.08300). Common Corpus: ~2T tokens (Pleias paper). Reddit IPO filing: $203M data contracts.
- Reported โ ๏ธ
- Crawl averages 2.69B / 2.53B / 2.15B per crawl (third-party analysis). Cloudflare default block 1 Jul 2025 and ~20% of sites (press). Ratios (Cloudflare Radar via press): Anthropic 286,930:1 (Jan 2025) โ 38,065:1 (Jul 2025); OpenAI ~1,091:1 (Jul 2025); Google ~14:1; inconsistent across reports. Common Crawl English share; Indian language programmes; some crawler and deal details.
- Our illustrations
- Sourcing ratings, crawler simulator, robots.txt matcher (simplified) and the licence price calculator.
- Crawling and restrictions
- Common Crawl: September 2025 crawl
- Consent in Crisis (Longpre et al.)
- RFC 9309: Robots Exclusion Protocol
- MIT Technology Review: Cloudflare blocks AI bots by default
- PPC Land: crawl-to-referral ratios ยท Search Engine Journal: inconsistent ratios
- Datasets
- StarCoder2 and The Stack v2
- Institutional Books 1.0
- Common Corpus
- Part 1: the data layer, mapped