Refining the Internet: The Data Engineers Who Turn Petabytes of Web Pages into Training Tokens
A frontier model is, above all, the data it was trained on. Getting from billions of raw web pages to a clean, deduplicated, safe, well-mixed set of tens of trillions of tokens is a large engineering operation in its own right. What Meta did to Llama 3's data is told in The Data Refinery. This part is about the platform and the people: the crawling, processing and quality engineers, the open tools they use, and the open datasets that show how it is done.
1. Where we are in the stack
The AI factory stack, by role
2. The roles: data infrastructure engineer and data-quality researcher
Data work splits between engineers who build the pipelines and researchers who decide what "good data" means. Thinking Machines Lab has advertised a web-crawling research engineer to build crawlers, deduplication and quality filtering for pre-training data at petabyte scale. Cohere's Senior Member of Technical Staff, Web Data owns extraction, parsing, deduplication and filtering over Common Crawl. Grass's Research Crawling Engineer targets millions to billions of URLs a day with terabyte-to-petabyte datasets; Magic advertised a Software Engineer, Pretraining Data for crawlers and deduplication across modalities (since closed).
Two roles, two skill profiles
Our reading of postings (1โ5). The day timelines are composite illustrations.
3. The data pipeline
Every pre-training data pipeline follows roughly the same stages, though each lab tunes them differently. Most raw web text does not survive: boilerplate, spam, duplicates and low-quality pages are removed. Click a stage.
From raw crawl to training tokens
Illustrative shares of data surviving each stage; real numbers vary widely by lab and dataset.
4. The tooling layer
The data platform stack
Three trends stand out. Processing has moved onto GPUs: NVIDIA's NeMo Curator runs exact (MD5) and fuzzy (MinHash plus locality-sensitive hashing) deduplication on GPUs, and NVIDIA says its semantic deduplication can remove up to 50% of large uncurated web datasets with minimal performance loss. Models now filter data: classifiers and LLMs score quality and topic, which turns data processing into a huge batch-inference job; Ray Data's LLM module runs vLLM or SGLang over datasets, and ByteDance reported embedding about 200 TB per run with a 10B+ model (a vendor case study). And open frameworks such as Hugging Face's datatrove make whole pipelines reproducible.
5. Open datasets as blueprints
Labs rarely publish their data pipelines, but open projects show the state of the art.
Three pipelines compared
DataComp-LM (DCLM), a NeurIPS 2024 benchmark, turned dataset design into a controlled experiment: a standardised 240-trillion-token pool from Common Crawl, fixed training recipes and 53 evaluations. Its DCLM-Baseline dataset trained a 7B model to 64% 5-shot MMLU with 2.6T tokens in the preprint (63% on 2T in the published version), 6.6 points better than MAP-Neo with 40% less compute. The lesson matches Meta's: model-based quality filtering is one of the biggest levers on model quality.
6. Deduplication at scale
The web is full of copies: mirrored pages, templates, quoted articles. Training on duplicates wastes compute and encourages memorisation. Exact duplicates are easy to catch with hashes. Near-duplicates need MinHash: break each document into overlapping word "shingles", hash them with many functions, and keep the minimum of each. The share of matching minimums estimates how similar two documents are, without comparing them word by word; locality-sensitive hashing then groups likely duplicates so billions of documents never need pairwise comparison. Semantic deduplication goes further, using embeddings to catch paraphrases.
MinHash demo
Two near-duplicate sentences. More hash functions give a closer estimate of the true Jaccard similarity.
7. Access, licensing and the shrinking web
The open web is getting harder to collect. Since 1 July 2025 Cloudflare blocks AI crawlers by default on new domains it serves, and many publishers block AI training crawlers in robots.txt. Blocking Common Crawl's crawler (CCBot) also removes a site from the Common Crawl dataset that many models build on. In response, labs have signed content-licensing deals with publishers and data providers and invested in synthetic data, a topic we will return to in a dedicated data series. For the data platform team, this adds new jobs: tracking provenance and licences per document, honouring opt-outs and removing data on request.
8. When it breaks
Data detective
Pick a symptom.
9. Handoffs and careers
Who the data platform team works with
- Typical backgrounds
- Data engineers from search and ads companies, web-crawling engineers, NLP researchers, big-data platform engineers.
- Core skills
- Spark/Ray/distributed processing, crawling and HTML extraction, hashing and deduplication, text classifiers and batch inference, data versioning, privacy and licensing awareness, careful evaluation of data changes.
- Growth path
- Data engineer โ owner of a pipeline stage (crawl, dedup, quality) โ lead for a lab's pre-training data mix.
- Further reading
- The Data Refinery ยท Part 4: Storage ยท How LLM Pretraining Works
๐๏ธ Case file: Meta (Llama 3)
- Scale
- About 15T tokens of training data.
- Deduplication
- URL, document and line level (lines appearing more than 6 times per 30M documents removed).
- Quality
- Model-based classifiers (DistilRoberta trained on Llama 2 annotations).
- Mix
- ~50% general knowledge, 25% maths and reasoning, 17% code, 8% multilingual. Full story in The Data Refinery.
๐ Next in the series
Part 9: Experiment and research tooling โ the research engineers and ML platform teams who build the tools for running, tracking and evaluating thousands of experiments.
๐ข Reading the numbers
- DCLM figures come from the paper (preprint: 64% on 2.6T tokens; published: 63% on 2T).
- FineWeb size and source come from Hugging Face; its pipeline steps come from a secondary summary โ ๏ธ.
- NeMo Curator capabilities and the "up to 50%" semantic-dedup figure come from NVIDIA's documentation; ByteDance's 200 TB is a vendor case study.
- Cloudflare's default blocking comes from reporting on its July 2025 change; licensing-deal values are omitted because we could not verify them.
- Role descriptions come from Thinking Machines Lab, Cohere, Grass and Magic postings.
- Funnel shares and the MinHash demo are illustrations.
๐ References
- Datasets and tools
- Li et al.: "DataComp-LM: In search of the next generation of training sets for language models" (NeurIPS 2024)
- FineWeb summary (secondary)
- NVIDIA NeMo Curator: deduplication concepts
- Ray Data: working with LLMs ยท ByteDance case study (Anyscale)
- Access
- BuzzStream: Which news sites block AI crawlers in 2025
- cside: CCBot and Common Crawl
- Roles
- Thinking Machines Lab: Web Crawling Research Engineer (posting)
- Cohere: Senior Member of Technical Staff, Web Data (posting)
- Grass / Wynd Labs: Research Crawling Engineer (posting)
- Related on this site
- The Data Refinery ยท Part 7: Training frameworks ยท Is It Ready? (contamination)