Home โ€บ Blog โ€บ Filtering and deduplication
Inside the Data Layer ยท Part 3 ยท Filtering

Throwing Away 90% of the Web: How AI Labs Filter and Deduplicate Training Data

Raw web crawls are mostly menus, spam, boilerplate and copies of copies. Before any of it reaches a GPU, a refinery of rules, classifiers and hashing algorithms decides what stays. The decisions are small (is this line too short? is this page too similar to another?) but they are made billions of times, and they shape what a model knows, what it memorises and how its benchmark scores should be read. This part opens up that refinery, with working versions of the heuristics, a quality filter and MinHash deduplication you can try.

FL
FrontierAGI Team
โš ๏ธ How this was researched (as of 27 Oct 2026) Sources are the papers for Gopher (quality rules), Lee et al. (deduplication), FineWeb and FineWeb-Edu, DCLM, Nemotron-CC, SemDeDup and the CCNet and RefinedWeb pipelines, plus tool documentation (datatrove, NeMo Curator). Figures quoted only by third parties are marked โš ๏ธ. The sample documents, rule lab and quality histogram are our own illustrations; the MinHash demo computes real (if small) signatures in your browser. See Reading the numbers.

1. Case file

๐Ÿงน
Filtering and deduplication, October 2026Stages 3โ€“5 of the data supply chain
Lee et al. (ACL 2022)One 61-word sentence 60,000+ times in C4
After deduplication~10ร— less memorised text emitted
FineWeb MinHash5-grams ยท 112 hashes ยท 14 bands of 8
FineWeb-Edu1.3T tokens kept by an LLM-trained classifier
DCLM-Baseline3.8T tokens via a fastText classifier
Model-based filteringremoves ~90% of data (Nemotron-CC)

2. Timeline

Cleaning the web, 1997 โ†’ 2025

Tap a milestone. Colour: rules, classifiers, deduplication, datasets.

Tap a milestone.

3. The refinery

Pipelines differ in detail but share an order: cheap checks first, expensive ones on what survives. Text extraction and language identification run on everything; rules remove obvious junk; classifiers score what is left; deduplication runs across the whole corpus; decontamination removes benchmark questions last.

Six documents through the pipeline

Step through the filters and watch documents drop out. Documents and verdicts are invented examples of real failure types.

4. Heuristic rules

The first serious filter is a set of hand-written rules. DeepMind's Gopher paper (2021) published an influential set, still reused in FineWeb, Dolma and NVIDIA's tools: keep documents of 50 to 100,000 words with a mean word length of 3 to 10 characters; drop those where the ratio of "#" or "โ€ฆ" symbols to words exceeds 0.1, more than 90% of lines start with a bullet, or more than 30% end with an ellipsis; require 80% of words to contain a letter and at least two of eight common English words ("the", "be", "to", "of", "and", "that", "have", "with"). C4 added rules of its own, such as dropping lines without terminal punctuation and pages containing "lorem ipsum" or curly braces.

Rule lab

Real Gopher-style checks run in your browser on a sample document. Pick a sample or type your own.

5. Quality classifiers

Rules catch junk but cannot judge whether text is worth learning from. Since 2023 the best datasets add a model-based quality filter. Three approaches dominate:

PerplexityKeep text that a small model trained on reference data (e.g. Wikipedia) finds predictable. Used by CCNet. Cheap, but rewards "Wikipedia-like" style.
fastText classifierA tiny linear classifier trained to separate "good" examples from random web text. DCLM used instruction data (OpenHermes 2.5) and top r/ExplainLikeImFive posts as positives, keeping 3.8T tokens.
LLM-annotated scoresAsk a large model to rate a sample of pages, then train a small classifier on its ratings to score everything. FineWeb-Edu used Llama-3-70B ratings of educational value (0โ€“5) on a few hundred thousand pages โš ๏ธ, keeping 1.3T tokens.

Where to set the threshold?

An illustrative distribution of educational scores over 15T tokens of filtered web text. Move the threshold: tokens kept fall fast, quality rises slowly. Our model, shaped like published FineWeb-Edu results, not their data.

The trade-off is the whole game. In a SmolLM2 comparison, FineWeb-Edu scored higher on knowledge benchmarks (MMLU, ARC, OpenBookQA) while DCLM did better on commonsense ones (HellaSwag, CommonsenseQA): each classifier encodes a view of "good" text. NVIDIA's Nemotron-CC therefore combined several classifiers into quality buckets and kept the lower tiers too (rephrasing some synthetically), because a 15-trillion-token run cannot afford to throw away 90% of the web.

6. Deduplication

The web repeats itself: syndicated news, mirrored sites, templates, licence texts, spam farms. Repetition wastes compute and makes models memorise. Lee et al. (ACL 2022) found one 61-word sentence over 60,000 times in C4, that over 1% of unprompted model output was copied verbatim from training data, and that over 4% of standard validation sets overlapped with training data. After deduplication, models emitted memorised text about ten times less often and reached the same accuracy in fewer steps.

Four kinds of duplicate removal

7. MinHash, hands on

Comparing every document to every other is impossible at billions of documents. MinHash (Broder, 1997) turns each document into a short signature: split it into overlapping word n-grams ("shingles"), hash each shingle with many different hash functions, and keep the minimum value for each. The fraction of matching minimums between two signatures estimates their Jaccard similarity (shared shingles รท all shingles).

MinHash calculator

Computed in your browser with real (small) hash functions. Edit either text.

8. Locality-sensitive hashing: the bands trick

Signatures are still too many to compare pairwise, so pipelines split each signature into b bands of r values and only compare documents that match exactly on at least one band. The chance that two documents with similarity s are caught is 1 โˆ’ (1 โˆ’ sr)b, an S-shaped curve whose steep part acts as the threshold. FineWeb used 112 hashes as 14 bands of 8, catching pairs at 0.75 similarity 77% of the time and at 0.85 about 99%.

LSH S-curve

๐Ÿ’ก FineWeb's surprise: deduplicate lessDeduplicating all 96 Common Crawl snapshots against each other removed up to 90% of the older snapshots, and the surviving text trained worse models: what remained after global dedup was disproportionately low-quality. Deduplicating each snapshot only against itself gave better results. More deduplication is not always better.

9. Decontamination

If benchmark questions sit in the training data, a model can score well by remembering answers. Decontamination searches the corpus for long n-gram overlaps with evaluation sets and removes matching documents (or, after the fact, flags affected benchmarks). It must run against every benchmark you plan to report, including new ones, and it cannot catch paraphrases. That is why evaluation relies increasingly on private and freshly written test sets.

10. Running it at scale

ToolWhoWhat it does
datatroveHugging FaceThe pipeline behind FineWeb: extraction, Gopher/C4 filters, MinHash dedup, on CPUs at cluster scale; open source
NeMo CuratorNVIDIAGPU-accelerated filtering, classifiers and fuzzy/semantic dedup; claimed ~16ร— faster fuzzy dedup on GPUs โš ๏ธ (third-party pages)
Dolma toolkitAI2Taggers, mixers and Bloom-filter dedup used for Dolma and OLMo
DataComp-LMAcademic consortiumA benchmark: fixed model and compute, compete only on data filtering
Google dedup toolsGoogle ResearchSuffix-array exact-substring deduplication from Lee et al.

11. What filters get wrong

12. How strict?

Pick a filtering strategy

An editorial guide.

VerdictFiltering is where data becomes a design choice. Rules remove the obvious junk; classifiers decide what "good" means and quietly shape a model's strengths; deduplication cuts memorisation but, done globally, can hurt; decontamination keeps evaluations honest. The best open pipelines publish every threshold. Frontier labs do not, but they face the same trade-off: filter harder for quality, or keep more tokens for scale.

Inside the Data Layer

13. Reading the numbers

Primary
Gopher rules: 50โ€“100,000 words, mean word length 3โ€“10, symbol ratio โ‰ค 0.1, bullets โ‰ค 90%, ellipsis lines โ‰ค 30%, 80% alphabetic words, 2 of 8 stop words (Rae et al., 2021). Lee et al.: 61-word sentence 60,000+ times in C4; >1% verbatim output; ~10ร— less memorisation; >4% validation overlap (ACL 2022). FineWeb: 5-grams, 112 hashes, 14ร—8 bands; 56/77/92/98.8% detection at 0.70/0.75/0.80/0.85; per-snapshot beat global dedup (paper). FineWeb-Edu 1.3T; DCLM 3.8T with OH2.5 + ELI5 fastText (model cards). SemDeDup: 50% of LAION removed with minimal loss (ICML 2023). Nemotron-CC ~90% removal remark (paper).
Secondary โš ๏ธ
FineWeb-Edu annotation count (~450Kโ€“500K) and keep threshold; NeMo Curator speedups; SmolLM2 benchmark comparison is one ablation.
Our illustrations
Sample documents and verdicts, quality histogram and benchmark curve, strategy chooser. The rule lab and MinHash demo compute real values on your text but are simplified.