Home โ€บ Blog โ€บ Crawling and sourcing
Inside the Data Layer ยท Part 2 ยท Sourcing

Where the Tokens Come From: Crawlers, Licences and the Closing Web

Every training run starts with a shopping list: web pages, code, books, papers, forums, news archives, transcripts. Some of it is crawled for free, some is bought, some is public domain, and a growing share is now blocked. This part explains how crawlers collect the web, what each source is worth, how websites are fighting back, and why labs increasingly pay for data they used to take.

FL
FrontierAGI Team
โš ๏ธ How this was researched (as of 27 Oct 2026) Sources are Common Crawl's crawl announcements, the "Consent in Crisis" paper (NeurIPS 2024), Cloudflare's announcements and Radar data as reported by the press, dataset papers and cards (The Stack v2, Common Corpus, Institutional Books), RFC 9309 and crawler documentation. Crawl-to-referral ratios are reported inconsistently and are marked โš ๏ธ. Ratings in the sourcing matrix and the crawler simulator are our own illustrations. Disclosure: Claude, made by Anthropic, helps write this site; Anthropic's crawler appears in this article alongside others, reported from public data.

1. Case file

๐Ÿ•ท๏ธ
Data sourcing, October 2026Stage 1โ€“2 of the data supply chain
Common Crawl, Sep 2025 crawl2.39B pages ยท 421 TiB ยท archive >10 PB
Pages per Common Crawl crawl~2.69B (2024) โ†’ ~2.53B (2025) โ†’ ~2.15B (2026) โš ๏ธ
Consent in Crisis (2023โ†’2024)5%+ of C4 tokens, 28%+ of top sources fully blocked
Cloudflare (1 Jul 2025)AI crawlers blocked by default; pay per crawl
Open code: The Stack v267.5 TB raw, ~900B training tokens, 600+ languages
Open books: Institutional Books983K public-domain volumes, 242B tokens

2. Timeline

Sourcing training data, 1994 โ†’ 2026

Tap a milestone. Colour: crawling, open datasets, restrictions, deals.

Tap a milestone.

No single source is enough. Labs blend many, each with a different mix of size, quality, legal risk and cost. The open web provides volume; code, books and papers provide reasoning and depth; forums provide conversation; licensed archives provide freshness, quality and legal cover.

Source ร— trade-off

Our editorial ratings (1 = low, 5 = high). Tap a row for details.

Tap a source.

4. How a crawler works

A crawler keeps a frontier, a queue of URLs to visit, seeded with known sites. For each URL it checks the site's robots.txt, waits a polite interval so it does not overload the server, fetches the page, stores the raw response (Common Crawl uses the WARC archive format), extracts new links and adds them to the frontier. At web scale this means billions of pages, deduplicated URLs, revisit schedules for pages that change, and sometimes a headless browser to render JavaScript.

Crawler simulator

A toy crawler over a small web. Choose which user-agent it identifies as, then step. Sites and rules are invented.

Frontier (queue)
Stored (WARC)
Skipped

5. Common Crawl: the commons everyone uses

Common Crawl, a non-profit founded in 2008, publishes a fresh crawl roughly every month and has archived more than 10 petabytes. Most open pre-training datasets (C4, RefinedWeb, FineWeb, DCLM, Nemotron-CC) are built from it, and many labs use it alongside their own crawlers. Its September 2025 crawl held 2.39 billion pages and 421 TiB of uncompressed content. Crawls have been getting smaller, which one analysis attributes partly to more sites blocking its crawler, CCBot โš ๏ธ.

Pages per crawl

Average pages per monthly crawl. 2024โ€“2026 averages from a third-party analysis of Common Crawl announcements (โš ๏ธ); Sep 2025 from Common Crawl.

6. robots.txt: the web's polite "no"

Since 1994, sites have used a plain-text file, robots.txt, to tell crawlers what they may fetch; it was finally standardised as RFC 9309 in 2022. It is a request, not a lock: well-behaved crawlers obey it, others ignore it. AI companies publish separate user-agents so sites can opt out of training without disappearing from search: OpenAI's GPTBot, Anthropic's ClaudeBot, Common Crawl's CCBot, and tokens such as Google-Extended and Applebot-Extended that control AI use of content without blocking the search crawler itself.

robots.txt explorer

Edit the rules or pick a preset, then see what each crawler may fetch. Simplified matching (longest prefix wins; Allow beats Disallow on ties).

Presets:

Google-Extended is a control token: Google's search crawler still fetches pages, but blocking it opts content out of Gemini training and grounding. Real crawlers also support wildcards; this matcher does not.

7. The closing web

The Data Provenance Initiative's "Consent in Crisis" study (NeurIPS 2024) audited about 14,000 web domains and found that in a single year (2023โ€“2024) robots.txt rules had fully restricted 5%+ of all tokens in C4, and 28%+ of its most actively maintained, critical sources; terms of service restricted 45% of C4. Then, on 1 July 2025, Cloudflare, which sits in front of roughly a fifth of websites, began blocking known AI crawlers by default for new customers and launched "pay per crawl", letting sites charge for access.

Consent in Crisis: C4 restrictions

From the paper. Note the units: tokens vs sources.

Crawl-to-referral ratios โš ๏ธ

Pages crawled for each visitor sent back, from Cloudflare Radar as reported by the press. Log scale. Figures vary widely by month and by report; Search Engine Journal found seven different published ratios for one company. Treat as orders of magnitude.

A search engine crawls pages and sends readers back. A training crawler takes pages and, until its product links to sources, sends almost no one back. Anthropic's reported ratio fell sharply after Claude gained web search with citations.

8. Buying instead of crawling

As the free web closes, labs buy access. Licences offer more than tokens: guaranteed freshness, high-quality archives behind paywalls, structured data, and legal cover. Reddit's 2024 IPO filing disclosed $203 million of data-licensing contracts over two to three years; News Corp's deal with OpenAI was reported at more than $250 million over five years. Part 1 lists more.

What does a licensed token cost?

Our own arithmetic. Pick a reported deal value and estimate how many usable tokens it buys; the price per million tokens is very sensitive to the guess.

9. Code, books and science up close

๐Ÿ’ป CodeThe Stack v2 (BigCode, 2024) is drawn from the Software Heritage archive: 67.5 TB raw, 32.1 TB deduplicated, about 900B training tokens, 600+ languages. Licence filtering and an opt-out process ("Am I in The Stack?") let developers remove their code. StarCoder2 models were trained on it.
๐Ÿ“š BooksBooks are long, carefully edited and valuable, and the most legally fraught (see Part 1). Institutional Books 1.0 (Harvard, June 2025) released 983,004 public-domain volumes scanned through Google Books, 242B tokens in over 250 languages, with OCR cleaned up.
๐Ÿ”ฌ SciencePapers carry equations, tables and reasoning, but mostly arrive as PDFs. Extraction tools convert layout and maths into text; errors here silently degrade a model's science. Open sources include arXiv and open-access repositories.
๐Ÿ›๏ธ Public domain & open licenceCommon Corpus (Pleias, 2024โ€“25) assembles about 2 trillion tokens of public-domain and permissively licensed text, mostly English and French, as an "ethical" alternative for training.

10. Languages

The web is mostly English and a handful of large languages, and crawls inherit that skew; under half of Common Crawl's pages are estimated to be English โš ๏ธ, but most other languages are thin and lower quality after filtering. Labs that care about other languages turn to national libraries, government corpora, news archives, translation and synthetic data. Governments increasingly fund this: India's language programmes (such as Bhashini and AI Kosh โš ๏ธ), Europe's multilingual projects and others aim to give their languages enough clean text to train on. Part 6 covers how language shares are set in the mix.

11. Who uses what

OrganisationOwn crawlerSources disclosedLicensing
OpenAIGPTBot (documented)LittleMany publisher deals (News Corp, AP and others) reported
AnthropicClaudeBot (documented)Little; respects robots.txt per its docsBook settlement; deals not widely reported โš ๏ธ
GoogleGooglebot + Google-Extended controlLittleReddit deal reported
MetaOwn crawler โš ๏ธLlama 3: public sources, ~15T tokensSome deals reported โš ๏ธ
AI2, Hugging Face, NVIDIAMostly Common CrawlFull pipelines publishedOpen data
BigCode, Pleias, HarvardSoftware Heritage, archivesFully documentedOpen licences, opt-outs

12. Choosing sources

What should your mix lean on?

An editorial guide.

VerdictThe free web is no longer free in practice. Crawls are shrinking, robots.txt and terms of service fence off the best-maintained sources, and infrastructure providers now block AI crawlers by default. The response is a split market: open datasets built from Common Crawl and public-domain archives for those who need provenance, and licensed, paywalled and private data for those who can pay. Sourcing has become as much a legal and commercial function as an engineering one.

Inside the Data Layer

13. Reading the numbers

Primary
Common Crawl Sep 2025: 2.39B pages, 421 TiB; archive >10 PB (Common Crawl). Consent in Crisis: ~14,000 domains; 5%+ of C4 tokens and 28%+ of critical sources fully restricted by robots.txt; 45% of C4 restricted by terms of service (NeurIPS 2024). The Stack v2: 67.5 TB raw, 32.1 TB deduplicated, ~900B tokens (BigCode/Hugging Face). Institutional Books: 983,004 volumes, 242B tokens (arXiv 2506.08300). Common Corpus: ~2T tokens (Pleias paper). Reddit IPO filing: $203M data contracts.
Reported โš ๏ธ
Crawl averages 2.69B / 2.53B / 2.15B per crawl (third-party analysis). Cloudflare default block 1 Jul 2025 and ~20% of sites (press). Ratios (Cloudflare Radar via press): Anthropic 286,930:1 (Jan 2025) โ†’ 38,065:1 (Jul 2025); OpenAI ~1,091:1 (Jul 2025); Google ~14:1; inconsistent across reports. Common Crawl English share; Indian language programmes; some crawler and deal details.
Our illustrations
Sourcing ratings, crawler simulator, robots.txt matcher (simplified) and the licence price calculator.