The Data Refinery: How 15 Trillion Tokens Were Collected, Cleaned and Mixed for Llama 3
Before a single GPU trains a frontier model, a data pipeline has to turn the messy public web into trillions of clean, deduplicated, carefully weighted tokens. For Llama 3, Meta assembled about 15 trillion, roughly eight times the 1.8 trillion behind Llama 2, and much of the improvement came from better data rather than more parameters. This part follows that refinery stage by stage: extraction, safety filters, deduplication, model-based quality scoring, the data mix, annealing and tokenization, plus the open pipelines anyone can study and the legal questions hanging over it all.
๐ Data, not just size
Part 1 mapped the software that runs an AI factory. But a training run is only as good as what it reads. Meta's paper describes building "more careful pre-processing and curation pipelines" for Llama 3's pre-training data, and data quality is one of the main differences it cites from Llama 2. The flagship 405B model was trained on 15.6 trillion text tokens.
This article covers the engineering of that corpus. For the concepts behind pre-training itself, see How LLM Pretraining Works.
๐ญ The refinery: from crawl to tokens
๐ง Click a stage
The typical stages of a frontier pre-training data pipeline, with what Llama 3 did where Meta described it.
๐ป How much survives? (illustrative)
Share of raw crawled text remaining after each stage, loosely based on published open pipelines; Meta did not publish its own figures.
Illustrative only. Open pipelines such as FineWeb and RefinedWeb report keeping only a small fraction of raw Common Crawl text after extraction, filtering and deduplication; exact shares vary widely by recipe.
๐งฝ Cleaning the web
- Extraction
- Web pages are mostly navigation, ads and markup. Meta built a custom HTML parser to pull out the main text, taking particular care with pages containing code and maths so equations and indentation survive. Open pipelines use tools like trafilatura; FineWeb found that extracting text itself beat Common Crawl's pre-extracted WET files.
- Safety & privacy
- Meta filtered out websites likely to contain unsafe content or high volumes of personal information (PII), domains ranked harmful under its safety standards, and known adult-content domains.
- Deduplication
- Three levels: identical URLs; global MinHash near-duplicate removal across the whole dataset; and line-level deduplication, removing any line that appeared more than six times in each bucket of 30 million documents. Meta noted this also removes some frequent high-quality text, but evaluations improved strongly.
- Heuristics
- Rule-based filters for low-quality pages: excessive repetition, very short or symbol-heavy text, "dirty word" counts and unusual token distributions, in the tradition of the Gopher and C4 rules.
๐งช Deduplication, demonstrated
๐ Three crawled pages
Toggle each step to see what gets removed. A toy example of how deduplication works, not Meta's code.
๐ Judging quality with models
Rules catch obvious junk; deciding whether a page is worth learning from takes models. Meta experimented with several classifiers:
- A fastText classifier trained to recognise whether a text would be referenced by Wikipedia.
- DistilRoberta classifiers, chosen for efficiency, scoring document quality; Meta's code and reasoning classifiers were DistilRoberta models trained on web data annotated by Llama 2.
- Per-language quality ranking for multilingual data, after language identification across 176 languages.
That recursion is now standard: each generation of models helps curate the data for the next. The open FineWeb-Edu dataset (1.3 trillion tokens) was filtered with a classifier trained on educational-value scores produced by Llama-3-70B-Instruct, and the DCLM baseline used a fastText classifier trained to tell instruction-style data (OpenHermes 2.5 and Reddit ELI5) apart from ordinary web text.
๐๏ธ Where do you set the bar?
A made-up distribution of quality scores across a web corpus. Move the cutoff: a stricter bar gives cleaner data but fewer tokens.
Illustrative. In practice labs tune cutoffs by training small models on each filtered set and comparing benchmark scores, and too strict a filter starves the model of diversity.
๐ฅฃ The mix
The final Llama 3 mix was roughly 50% general knowledge, 25% mathematics and reasoning, 17% code and 8% multilingual tokens. Meta set it with two tools: a knowledge classifier that sorts web content by type, used to downsample categories over-represented on the web (such as arts and entertainment), and scaling-law experiments, training several small models on a candidate mix and using them to predict how a large model would perform on it. The mix also changed during training: Meta increased non-English data, upsampled mathematics, added recent web data later in training and downsampled subsets judged lower quality.
๐๏ธ Design a data mix (illustrative)
Shares are normalised to 100%. The skill bars use a simple diminishing-returns toy model, not real scaling laws.
๐ฅ The final stages: long context and annealing
Llama 3's pre-training ran in three stages: initial pre-training, long-context pre-training (gradually extending the context window to 128K tokens) and annealing. During annealing, over roughly the final 40 million tokens per a survey of mid-training, the learning rate is brought down to zero while the mix is shifted toward the highest-quality sources. Meta also used short annealing runs as a cheap way to measure the value of small specialised datasets before including them. Part 5 covers the run itself.
๐ค Tokenization
Text becomes numbers through a tokenizer. Llama 3 uses a vocabulary of about 128,000 tokens, combining 100,000 tokens from OpenAI's tiktoken tokenizer with 28,000 extra tokens for non-English languages. Secondary summaries report that this raised compression on English text from about 3.17 to 3.94 characters per token versus Llama 2, so the same compute processes more text. See Tokenization & Byte-Pair Encoding for how such tokenizers are built.
โ๏ธ Tokenizer toy
Type a sentence. This is a crude approximation of subword splitting, not the real Llama 3 tokenizer.
๐งฐ Doing it at petabyte scale
Meta did not release its data or pipeline code, but open projects show how the work is done.
๐ Open pipelines and tools
| Project | From | Scale | Notable choices |
|---|---|---|---|
| FineWeb / datatrove | Hugging Face (2024) | ~15T tokens from Common Crawl | trafilatura extraction; MinHash dedup per crawl (global dedup hurt in ablations); full code and ablation models released |
| FineWeb-Edu | Hugging Face | ~1.3T tokens | Classifier trained on Llama-3-70B-Instruct educational-quality annotations |
| DCLM | DataComp consortium (2024) | 240T-token raw pool | fastText filter (OpenHermes + ELI5 vs web); 7B baseline reached 64% MMLU on 2.6T tokens |
| Dolma, RedPajama, RefinedWeb, The Pile, C4 | AI2, Together, TII, EleutherAI, Google | Hundreds of billions to trillions of tokens | Earlier open corpora that shaped today's recipes |
FineWeb and DCLM papers (2024). Distributed processing typically runs on Spark, Ray or Slurm-based workers, with Parquet storage and pre-tokenised shards written to object storage or a parallel file system.
The engineering challenge is throughput: processing petabytes of raw HTML means running extraction, language ID, classifiers and MinHash across thousands of CPU cores (GPUs for the larger classifiers), with checkpointable jobs so a failure does not restart a week of work. The output is a set of tokenised shards that the training job (Part 1) streams from storage.
โ๏ธ Data and the law
Training data is also a legal question. In Kadrey v. Meta, authors sued over Meta's use of books downloaded from "shadow libraries" to train Llama. In June 2025 a federal judge ruled that Meta's use was fair use, but narrowly: the decision turned on the plaintiffs' failure to build the right record, and the judge pointed to "market dilution" as a theory of harm that could succeed in future cases. Other lawsuits, licensing deals with publishers and websites, and robots.txt opt-outs continue to shape what data labs can use.
There is also a supply question. Epoch AI has estimated the stock of high-quality public human text at around 300 trillion tokens and projected it could be fully used by frontier training runs sometime between 2026 and 2032, one reason labs are investing in synthetic data, rephrasing and licensed sources.
๐ฐ๏ธ Timeline
๐ญ What to watch
- Synthetic data share: how much of the next generation's corpora is written by models.
- Data exhaustion: whether the 2026โ2032 window for public human text holds.
- Copyright cases and licensing deals that change what can be trained on.
- Open recipes: successors to FineWeb and DCLM that close the gap with closed labs.
- Next in this series: Part 3 plans the run itself: model size, token budget, scaling laws and the compute bill.
๐ข Reading the numbers
- Llama 3 figures (15T and 15.6T tokens, 50/25/17/8 mix, 176 languages, line dedup above six occurrences per 30M documents) come from Meta's paper via search snippets and summaries.
- The 40M-token annealing figure comes from a survey of mid-training, not the Llama 3 paper directly.
- Tokenizer compression (3.17 โ 3.94 characters per token) is from secondary summaries.
- Meta's data sources and per-stage volumes are not public; the funnel and toys are illustrative.
- Epoch AI's 300T-token estimate and 2026โ2032 window come from its published data-exhaustion study.
๐ References and papers
- Llama 3
- Llama Team, Meta: "The Llama 3 Herd of Models" (2024)
- Meta AI: Introducing Meta Llama 3
- Llama 3 model card
- Johns Hopkins course presentation on the Llama 3 paper
- "Mid-Training of Large Language Models: A Survey" (2025)
- Data pipelines and quality
- Penedo et al.: "The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale" (2024)
- Li et al.: "DataComp-LM: In search of the next generation of training sets for language models" (2024)
- Penedo et al.: "The RefinedWeb Dataset for Falcon LLM" (2023)
- Wenzek et al.: "CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data" (2019)
- Rae et al.: "Scaling Language Models: Methods, Analysis & Insights from Training Gopher" (2021)
- Lee et al.: "Deduplicating Training Data Makes Language Models Better" (2022)
- Soldaini et al.: "Dolma: an Open Corpus of Three Trillion Tokens" (2024)
- Gao et al.: "The Pile" (2020)
- Villalobos et al. (Epoch AI): "Will we run out of data? Limits of LLM scaling based on human-generated data"
- Law
- Loeb & Loeb: Kadrey v. Meta Platforms (2025)
- Maginative: Court sides with Meta on training with books
- Related on this site
- Part 1: The Software Stack of an AI Factory ยท How LLM Pretraining Works ยท Retrieval & the RAG Stack ยท The Llama Lineage