Throwing Away 90% of the Web: How AI Labs Filter and Deduplicate Training Data
Raw web crawls are mostly menus, spam, boilerplate and copies of copies. Before any of it reaches a GPU, a refinery of rules, classifiers and hashing algorithms decides what stays. The decisions are small (is this line too short? is this page too similar to another?) but they are made billions of times, and they shape what a model knows, what it memorises and how its benchmark scores should be read. This part opens up that refinery, with working versions of the heuristics, a quality filter and MinHash deduplication you can try.
1. Case file
2. Timeline
Cleaning the web, 1997 โ 2025
Tap a milestone. Colour: rules, classifiers, deduplication, datasets.
3. The refinery
Pipelines differ in detail but share an order: cheap checks first, expensive ones on what survives. Text extraction and language identification run on everything; rules remove obvious junk; classifiers score what is left; deduplication runs across the whole corpus; decontamination removes benchmark questions last.
Six documents through the pipeline
Step through the filters and watch documents drop out. Documents and verdicts are invented examples of real failure types.
4. Heuristic rules
The first serious filter is a set of hand-written rules. DeepMind's Gopher paper (2021) published an influential set, still reused in FineWeb, Dolma and NVIDIA's tools: keep documents of 50 to 100,000 words with a mean word length of 3 to 10 characters; drop those where the ratio of "#" or "โฆ" symbols to words exceeds 0.1, more than 90% of lines start with a bullet, or more than 30% end with an ellipsis; require 80% of words to contain a letter and at least two of eight common English words ("the", "be", "to", "of", "and", "that", "have", "with"). C4 added rules of its own, such as dropping lines without terminal punctuation and pages containing "lorem ipsum" or curly braces.
Rule lab
Real Gopher-style checks run in your browser on a sample document. Pick a sample or type your own.
5. Quality classifiers
Rules catch junk but cannot judge whether text is worth learning from. Since 2023 the best datasets add a model-based quality filter. Three approaches dominate:
Where to set the threshold?
An illustrative distribution of educational scores over 15T tokens of filtered web text. Move the threshold: tokens kept fall fast, quality rises slowly. Our model, shaped like published FineWeb-Edu results, not their data.
The trade-off is the whole game. In a SmolLM2 comparison, FineWeb-Edu scored higher on knowledge benchmarks (MMLU, ARC, OpenBookQA) while DCLM did better on commonsense ones (HellaSwag, CommonsenseQA): each classifier encodes a view of "good" text. NVIDIA's Nemotron-CC therefore combined several classifiers into quality buckets and kept the lower tiers too (rephrasing some synthetically), because a 15-trillion-token run cannot afford to throw away 90% of the web.
6. Deduplication
The web repeats itself: syndicated news, mirrored sites, templates, licence texts, spam farms. Repetition wastes compute and makes models memorise. Lee et al. (ACL 2022) found one 61-word sentence over 60,000 times in C4, that over 1% of unprompted model output was copied verbatim from training data, and that over 4% of standard validation sets overlapped with training data. After deduplication, models emitted memorised text about ten times less often and reached the same accuracy in fewer steps.
Four kinds of duplicate removal
7. MinHash, hands on
Comparing every document to every other is impossible at billions of documents. MinHash (Broder, 1997) turns each document into a short signature: split it into overlapping word n-grams ("shingles"), hash each shingle with many different hash functions, and keep the minimum value for each. The fraction of matching minimums between two signatures estimates their Jaccard similarity (shared shingles รท all shingles).
MinHash calculator
Computed in your browser with real (small) hash functions. Edit either text.
8. Locality-sensitive hashing: the bands trick
Signatures are still too many to compare pairwise, so pipelines split each signature into b bands of r values and only compare documents that match exactly on at least one band. The chance that two documents with similarity s are caught is 1 โ (1 โ sr)b, an S-shaped curve whose steep part acts as the threshold. FineWeb used 112 hashes as 14 bands of 8, catching pairs at 0.75 similarity 77% of the time and at 0.85 about 99%.
LSH S-curve
9. Decontamination
If benchmark questions sit in the training data, a model can score well by remembering answers. Decontamination searches the corpus for long n-gram overlaps with evaluation sets and removes matching documents (or, after the fact, flags affected benchmarks). It must run against every benchmark you plan to report, including new ones, and it cannot catch paraphrases. That is why evaluation relies increasingly on private and freshly written test sets.
10. Running it at scale
| Tool | Who | What it does |
|---|---|---|
| datatrove | Hugging Face | The pipeline behind FineWeb: extraction, Gopher/C4 filters, MinHash dedup, on CPUs at cluster scale; open source |
| NeMo Curator | NVIDIA | GPU-accelerated filtering, classifiers and fuzzy/semantic dedup; claimed ~16ร faster fuzzy dedup on GPUs โ ๏ธ (third-party pages) |
| Dolma toolkit | AI2 | Taggers, mixers and Bloom-filter dedup used for Dolma and OLMo |
| DataComp-LM | Academic consortium | A benchmark: fixed model and compute, compete only on data filtering |
| Google dedup tools | Google Research | Suffix-array exact-substring deduplication from Lee et al. |
11. What filters get wrong
- Dialects and languages: English-tuned rules (stop words, word lengths) and classifiers penalise other languages and non-standard English. FineWeb2 had to re-tune its pipeline per language.
- Style over substance: classifiers trained on textbooks or ELI5 answers reward a style; useful text that looks different (forums, code comments, poetry) is thrown away.
- Topic erasure: toxicity filters can remove discussion of health, sexuality or minority identities.
- Running out: aggressive filters leave too few tokens for long runs, forcing repetition or synthetic rephrasing (Part 4).
12. How strict?
Pick a filtering strategy
An editorial guide.
Inside the Data Layer
13. Reading the numbers
- Primary
- Gopher rules: 50โ100,000 words, mean word length 3โ10, symbol ratio โค 0.1, bullets โค 90%, ellipsis lines โค 30%, 80% alphabetic words, 2 of 8 stop words (Rae et al., 2021). Lee et al.: 61-word sentence 60,000+ times in C4; >1% verbatim output; ~10ร less memorisation; >4% validation overlap (ACL 2022). FineWeb: 5-grams, 112 hashes, 14ร8 bands; 56/77/92/98.8% detection at 0.70/0.75/0.80/0.85; per-snapshot beat global dedup (paper). FineWeb-Edu 1.3T; DCLM 3.8T with OH2.5 + ELI5 fastText (model cards). SemDeDup: 50% of LAION removed with minimal loss (ICML 2023). Nemotron-CC ~90% removal remark (paper).
- Secondary โ ๏ธ
- FineWeb-Edu annotation count (~450Kโ500K) and keep threshold; NeMo Curator speedups; SmolLM2 benchmark comparison is one ablation.
- Our illustrations
- Sample documents and verdicts, quality histogram and benchmark curve, strategy chooser. The rule lab and MinHash demo compute real values on your text but are simplified.
- Papers
- Gopher (Rae et al., 2021)
- Deduplicating training data makes language models better
- FineWeb and FineWeb-Edu ยท FineWeb2
- DCLM ยท fasttext-oh-eli5 model card
- Nemotron-CC ยท SmolLM2
- SemDeDup ยท CCNet
- Tools
- datatrove ยท NeMo Curator ยท Google deduplicate-text-datasets
- Part 2: crawling and sourcing ยท Part 1: the map