Home โ€บ Blog โ€บ The Data Refinery
Producing Intelligence ยท Part 2 ยท Data

The Data Refinery: How 15 Trillion Tokens Were Collected, Cleaned and Mixed for Llama 3

Before a single GPU trains a frontier model, a data pipeline has to turn the messy public web into trillions of clean, deduplicated, carefully weighted tokens. For Llama 3, Meta assembled about 15 trillion, roughly eight times the 1.8 trillion behind Llama 2, and much of the improvement came from better data rather than more parameters. This part follows that refinery stage by stage: extraction, safety filters, deduplication, model-based quality scoring, the data mix, annealing and tokenization, plus the open pipelines anyone can study and the legal questions hanging over it all.

FL
FrontierAGI Team
โš ๏ธ How this was researched Llama 3 facts come from Meta's paper "The Llama 3 Herd of Models" (2024) as quoted in search results, a Johns Hopkins course presentation on it, and Meta's launch blog; we could not open the full paper text directly. Meta did not publish its data sources, per-stage volumes or the datasets themselves, so the funnel, deduplication demo, quality slider, mix designer and tokenizer toy are illustrations, not Meta's numbers. The tokenizer compression figure comes from secondary summaries. Open-pipeline facts come from the FineWeb and DCLM papers. See Reading the numbers.

๐Ÿ“š Data, not just size

Part 1 mapped the software that runs an AI factory. But a training run is only as good as what it reads. Meta's paper describes building "more careful pre-processing and curation pipelines" for Llama 3's pre-training data, and data quality is one of the main differences it cites from Llama 2. The flagship 405B model was trained on 15.6 trillion text tokens.

~15Ttokens in Llama 3's pre-training corpus (405B saw 15.6T)
~8ร—more than Llama 2's 1.8T tokens
176languages sorted by a fastText language-ID model
50/25/17/8% general knowledge / maths & reasoning / code / multilingual

This article covers the engineering of that corpus. For the concepts behind pre-training itself, see How LLM Pretraining Works.

๐Ÿญ The refinery: from crawl to tokens

๐Ÿ”ง Click a stage

The typical stages of a frontier pre-training data pipeline, with what Llama 3 did where Meta described it.

๐Ÿ”ป How much survives? (illustrative)

Share of raw crawled text remaining after each stage, loosely based on published open pipelines; Meta did not publish its own figures.

Illustrative only. Open pipelines such as FineWeb and RefinedWeb report keeping only a small fraction of raw Common Crawl text after extraction, filtering and deduplication; exact shares vary widely by recipe.

๐Ÿงฝ Cleaning the web

Extraction
Web pages are mostly navigation, ads and markup. Meta built a custom HTML parser to pull out the main text, taking particular care with pages containing code and maths so equations and indentation survive. Open pipelines use tools like trafilatura; FineWeb found that extracting text itself beat Common Crawl's pre-extracted WET files.
Safety & privacy
Meta filtered out websites likely to contain unsafe content or high volumes of personal information (PII), domains ranked harmful under its safety standards, and known adult-content domains.
Deduplication
Three levels: identical URLs; global MinHash near-duplicate removal across the whole dataset; and line-level deduplication, removing any line that appeared more than six times in each bucket of 30 million documents. Meta noted this also removes some frequent high-quality text, but evaluations improved strongly.
Heuristics
Rule-based filters for low-quality pages: excessive repetition, very short or symbol-heavy text, "dirty word" counts and unusual token distributions, in the tradition of the Gopher and C4 rules.

๐Ÿงช Deduplication, demonstrated

๐Ÿ” Three crawled pages

Toggle each step to see what gets removed. A toy example of how deduplication works, not Meta's code.

๐Ÿ… Judging quality with models

Rules catch obvious junk; deciding whether a page is worth learning from takes models. Meta experimented with several classifiers:

That recursion is now standard: each generation of models helps curate the data for the next. The open FineWeb-Edu dataset (1.3 trillion tokens) was filtered with a classifier trained on educational-value scores produced by Llama-3-70B-Instruct, and the DCLM baseline used a fastText classifier trained to tell instruction-style data (OpenHermes 2.5 and Reddit ELI5) apart from ordinary web text.

๐ŸŽš๏ธ Where do you set the bar?

A made-up distribution of quality scores across a web corpus. Move the cutoff: a stricter bar gives cleaner data but fewer tokens.

Illustrative. In practice labs tune cutoffs by training small models on each filtered set and comparing benchmark scores, and too strict a filter starves the model of diversity.

๐Ÿฅฃ The mix

The final Llama 3 mix was roughly 50% general knowledge, 25% mathematics and reasoning, 17% code and 8% multilingual tokens. Meta set it with two tools: a knowledge classifier that sorts web content by type, used to downsample categories over-represented on the web (such as arts and entertainment), and scaling-law experiments, training several small models on a candidate mix and using them to predict how a large model would perform on it. The mix also changed during training: Meta increased non-English data, upsampled mathematics, added recent web data later in training and downsampled subsets judged lower quality.

๐ŸŽ›๏ธ Design a data mix (illustrative)

Shares are normalised to 100%. The skill bars use a simple diminishing-returns toy model, not real scaling laws.

๐Ÿ”ฅ The final stages: long context and annealing

Llama 3's pre-training ran in three stages: initial pre-training, long-context pre-training (gradually extending the context window to 128K tokens) and annealing. During annealing, over roughly the final 40 million tokens per a survey of mid-training, the learning rate is brought down to zero while the mix is shifted toward the highest-quality sources. Meta also used short annealing runs as a cheap way to measure the value of small specialised datasets before including them. Part 5 covers the run itself.

๐Ÿ”ค Tokenization

Text becomes numbers through a tokenizer. Llama 3 uses a vocabulary of about 128,000 tokens, combining 100,000 tokens from OpenAI's tiktoken tokenizer with 28,000 extra tokens for non-English languages. Secondary summaries report that this raised compression on English text from about 3.17 to 3.94 characters per token versus Llama 2, so the same compute processes more text. See Tokenization & Byte-Pair Encoding for how such tokenizers are built.

โœ‚๏ธ Tokenizer toy

Type a sentence. This is a crude approximation of subword splitting, not the real Llama 3 tokenizer.

๐Ÿงฐ Doing it at petabyte scale

Meta did not release its data or pipeline code, but open projects show how the work is done.

๐Ÿ“‹ Open pipelines and tools

ProjectFromScaleNotable choices
FineWeb / datatroveHugging Face (2024)~15T tokens from Common Crawltrafilatura extraction; MinHash dedup per crawl (global dedup hurt in ablations); full code and ablation models released
FineWeb-EduHugging Face~1.3T tokensClassifier trained on Llama-3-70B-Instruct educational-quality annotations
DCLMDataComp consortium (2024)240T-token raw poolfastText filter (OpenHermes + ELI5 vs web); 7B baseline reached 64% MMLU on 2.6T tokens
Dolma, RedPajama, RefinedWeb, The Pile, C4AI2, Together, TII, EleutherAI, GoogleHundreds of billions to trillions of tokensEarlier open corpora that shaped today's recipes

FineWeb and DCLM papers (2024). Distributed processing typically runs on Spark, Ray or Slurm-based workers, with Parquet storage and pre-tokenised shards written to object storage or a parallel file system.

The engineering challenge is throughput: processing petabytes of raw HTML means running extraction, language ID, classifiers and MinHash across thousands of CPU cores (GPUs for the larger classifiers), with checkpointable jobs so a failure does not restart a week of work. The output is a set of tokenised shards that the training job (Part 1) streams from storage.

โš–๏ธ Data and the law

Training data is also a legal question. In Kadrey v. Meta, authors sued over Meta's use of books downloaded from "shadow libraries" to train Llama. In June 2025 a federal judge ruled that Meta's use was fair use, but narrowly: the decision turned on the plaintiffs' failure to build the right record, and the judge pointed to "market dilution" as a theory of harm that could succeed in future cases. Other lawsuits, licensing deals with publishers and websites, and robots.txt opt-outs continue to shape what data labs can use.

There is also a supply question. Epoch AI has estimated the stock of high-quality public human text at around 300 trillion tokens and projected it could be fully used by frontier training runs sometime between 2026 and 2032, one reason labs are investing in synthetic data, rephrasing and licensed sources.

๐Ÿ•ฐ๏ธ Timeline

2019 ยท Google's C4 dataset (with T5); Meta's CCNet pipeline for filtering Common Crawl.
2020 ยท EleutherAI's The Pile.
2021 ยท DeepMind's Gopher paper popularises rule-based quality filters.
2022 ยท Research shows deduplication improves models; Epoch AI first warns high-quality text could run out.
2023 ยท RefinedWeb (Falcon) shows filtered web data alone can rival curated corpora; Llama 2 trained on 1.8T tokens.
2024 ยท AI2's Dolma; Llama 3 (April, 405B in July) on ~15T tokens; FineWeb, FineWeb-Edu and DCLM released.
Jun 2025 ยท Kadrey v. Meta: Meta's book training ruled fair use for these plaintiffs.
2025โ€“26 ยท Synthetic data, rephrased web text and licensing deals grow as sources.

๐Ÿ”ญ What to watch

๐ŸŽฏ The bottom line A frontier model's knowledge is refined, not just collected. For Llama 3, Meta turned the web and other sources into about 15 trillion tokens through custom extraction, safety filters, three levels of deduplication, model-based quality classifiers trained with help from Llama 2, a mix set by knowledge classification and small-model scaling experiments, a final annealing stage on the best data, and a larger tokenizer. Most of that pipeline is unglamorous data engineering at petabyte scale, and it explains a large share of the jump from Llama 2 to Llama 3.

๐Ÿ”ข Reading the numbers

๐Ÿ“š References and papers