Home โ€บ Blog โ€บ The Data Platform
Who Runs the AI Factory ยท Part 8 ยท Data platform

Refining the Internet: The Data Engineers Who Turn Petabytes of Web Pages into Training Tokens

A frontier model is, above all, the data it was trained on. Getting from billions of raw web pages to a clean, deduplicated, safe, well-mixed set of tens of trillions of tokens is a large engineering operation in its own right. What Meta did to Llama 3's data is told in The Data Refinery. This part is about the platform and the people: the crawling, processing and quality engineers, the open tools they use, and the open datasets that show how it is done.

FL
FrontierAGI Team
โš ๏ธ How this was researched Sources are the DataComp-LM paper (NeurIPS 2024), coverage of Hugging Face's FineWeb (some pipeline details from a secondary summary, marked โš ๏ธ secondary), NVIDIA NeMo Curator and Ray documentation, reporting on AI-crawler blocking, and real job postings (no Meta, OpenAI or Anthropic data postings were found). The funnel, MinHash demo and pipeline shares are illustrations. See Reading the numbers.

1. Where we are in the stack

The AI factory stack, by role

2. The roles: data infrastructure engineer and data-quality researcher

Data work splits between engineers who build the pipelines and researchers who decide what "good data" means. Thinking Machines Lab has advertised a web-crawling research engineer to build crawlers, deduplication and quality filtering for pre-training data at petabyte scale. Cohere's Senior Member of Technical Staff, Web Data owns extraction, parsing, deduplication and filtering over Common Crawl. Grass's Research Crawling Engineer targets millions to billions of URLs a day with terabyte-to-petabyte datasets; Magic advertised a Software Engineer, Pretraining Data for crawlers and deduplication across modalities (since closed).

Two roles, two skill profiles

Our reading of postings (1โ€“5). The day timelines are composite illustrations.

3. The data pipeline

Every pre-training data pipeline follows roughly the same stages, though each lab tunes them differently. Most raw web text does not survive: boilerplate, spam, duplicates and low-quality pages are removed. Click a stage.

From raw crawl to training tokens

Illustrative shares of data surviving each stage; real numbers vary widely by lab and dataset.

4. The tooling layer

The data platform stack

Three trends stand out. Processing has moved onto GPUs: NVIDIA's NeMo Curator runs exact (MD5) and fuzzy (MinHash plus locality-sensitive hashing) deduplication on GPUs, and NVIDIA says its semantic deduplication can remove up to 50% of large uncurated web datasets with minimal performance loss. Models now filter data: classifiers and LLMs score quality and topic, which turns data processing into a huge batch-inference job; Ray Data's LLM module runs vLLM or SGLang over datasets, and ByteDance reported embedding about 200 TB per run with a 10B+ model (a vendor case study). And open frameworks such as Hugging Face's datatrove make whole pipelines reproducible.

5. Open datasets as blueprints

Labs rarely publish their data pipelines, but open projects show the state of the art.

Three pipelines compared

DataComp-LM (DCLM), a NeurIPS 2024 benchmark, turned dataset design into a controlled experiment: a standardised 240-trillion-token pool from Common Crawl, fixed training recipes and 53 evaluations. Its DCLM-Baseline dataset trained a 7B model to 64% 5-shot MMLU with 2.6T tokens in the preprint (63% on 2T in the published version), 6.6 points better than MAP-Neo with 40% less compute. The lesson matches Meta's: model-based quality filtering is one of the biggest levers on model quality.

6. Deduplication at scale

The web is full of copies: mirrored pages, templates, quoted articles. Training on duplicates wastes compute and encourages memorisation. Exact duplicates are easy to catch with hashes. Near-duplicates need MinHash: break each document into overlapping word "shingles", hash them with many functions, and keep the minimum of each. The share of matching minimums estimates how similar two documents are, without comparing them word by word; locality-sensitive hashing then groups likely duplicates so billions of documents never need pairwise comparison. Semantic deduplication goes further, using embeddings to catch paraphrases.

MinHash demo

Two near-duplicate sentences. More hash functions give a closer estimate of the true Jaccard similarity.

Document A

Document B

7. Access, licensing and the shrinking web

The open web is getting harder to collect. Since 1 July 2025 Cloudflare blocks AI crawlers by default on new domains it serves, and many publishers block AI training crawlers in robots.txt. Blocking Common Crawl's crawler (CCBot) also removes a site from the Common Crawl dataset that many models build on. In response, labs have signed content-licensing deals with publishers and data providers and invested in synthetic data, a topic we will return to in a dedicated data series. For the data platform team, this adds new jobs: tracking provenance and licences per document, honouring opt-outs and removing data on request.

8. When it breaks

Data detective

Pick a symptom.

9. Handoffs and careers

Who the data platform team works with

๐Ÿงญ Getting into the role
Typical backgrounds
Data engineers from search and ads companies, web-crawling engineers, NLP researchers, big-data platform engineers.
Core skills
Spark/Ray/distributed processing, crawling and HTML extraction, hashing and deduplication, text classifiers and batch inference, data versioning, privacy and licensing awareness, careful evaluation of data changes.
Growth path
Data engineer โ†’ owner of a pipeline stage (crawl, dedup, quality) โ†’ lead for a lab's pre-training data mix.
Further reading
The Data Refinery ยท Part 4: Storage ยท How LLM Pretraining Works

๐Ÿ—‚๏ธ Case file: Meta (Llama 3)

Scale
About 15T tokens of training data.
Deduplication
URL, document and line level (lines appearing more than 6 times per 30M documents removed).
Quality
Model-based classifiers (DistilRoberta trained on Llama 2 annotations).
Mix
~50% general knowledge, 25% maths and reasoning, 17% code, 8% multilingual. Full story in The Data Refinery.
Verdict Data platforms decide model quality as much as any architecture choice, and they are now serious distributed systems: GPU-accelerated deduplication, model-based filtering running as massive batch inference, and provenance tracking for a web that is closing its doors. Open projects like FineWeb and DCLM have made the methods public; the competitive edge now lies in scale, judgement about quality, and access to data others cannot get.

๐Ÿ‘€ Next in the series

Part 9: Experiment and research tooling โ€” the research engineers and ML platform teams who build the tools for running, tracking and evaluating thousands of experiments.

๐Ÿ”ข Reading the numbers

๐Ÿ“š References