The Data Refinery
Phase 2 starts with what the machine eats. Maestro-1 needs about 10 trillion tokens (Part 3), and the quality of those tokens matters as much as the size of the model. This part builds the data refinery that turns raw web pages, code and partner tasks into a training mixture: where the data comes from and what Gopala may use, three filters with runnable code, deduplication, the mixture plan, how verified skills enter the data, and the records that let him answer "what did the model see?" a year later.
quality_filter.py, minhash_dedup.py and mixture.py were run (real output below) on documents and token counts written for this article; available-token figures are our planning assumptions. pipeline.py uses an external library and was not executed for this article. Gopala is a teaching scenario. Disclosure: Claude, made by Anthropic, helps write this site.
1. Mission log: weeks 14โ36
- Lists every candidate source with its licence, terms and opt-out status; drops anything he cannot defend to a partner's lawyer.
- Downloads public filtered web datasets as a starting point instead of crawling everything himself.
- Builds the filter โ dedup โ classify pipeline on the pre-emptible partition (Part 7), using spare GPU and CPU hours between Maestro-0 jobs.
- Writes the three scripts in this part and runs them on samples by hand before trusting them at scale.
- Runs small "data ablations" (Part 9) to choose the mixture; freezes the 10T-token blend for the main run.
- Every shard gets a manifest entry: source, licence, filters applied, hash.
2. Why data decides the model
- Same compute, different data, very different models.
- Hugging Face's FineWeb work showed that careful filtering and deduplication of the same Common Crawl source produced better models than earlier web datasets at equal token counts.
- FineWeb-Edu, a 1.3T-token subset kept by an educational-quality classifier, did better again on knowledge and reasoning benchmarks.
- Frontier labs say the same.
- Meta's Llama 3 paper describes heavy investment in filtering, deduplication and quality classifiers, and a final mix of roughly 50% general knowledge, 25% maths and reasoning, 17% code and 8% multilingual tokens.
- Its 405B model was trained on about 15T tokens.
- For Gopala the data is also the moat.
- Part 1's thesis: verified skills from real partner work are the one ingredient competitors cannot download.
- The refinery is where that ingredient is cleaned, protected and blended in.
Where 10 trillion tokens come from (planned shares)
Our planned mixture for Maestro-1 (see mixture.py below). Llama 3's published mix is shown in the text for comparison.
3. What he may use
- Every source gets a row in a register before a single byte is trained on.
- Licence or terms of use.
- Whether the site's robots.txt or an opt-out signal excludes AI training.
- Personal data risk.
- Whether the source could contain partners' confidential code.
- The rule: if he could not explain a source to a partner's lawyer, it does not go in.
- Court cases and settlements over training data (see the site's law and the data wall article) make provenance a business risk, not only an ethical one.
- Code is the most delicate source.
- Only permissively licensed code, with licence detection per file.
- Partner code is never pre-training data; partners contribute verified skills derived from tasks, under contract (Part 3).
| Source | Use? | Why |
|---|---|---|
| Public filtered web datasets (e.g. FineWeb) | Yes, as a base | Documented pipeline, published licence terms, opt-outs honoured upstream |
| Own crawl of selected domains | Yes, limited | Respect robots.txt and opt-outs; identify the crawler |
| Permissively licensed code | Yes | Licence detected per file; copyleft excluded |
| Paywalled news and books without a licence | No | Legal risk outweighs value |
| Partner repositories | No | Only verified skills derived from tasks, under contract |
| Synthetic data from his own models | Yes | Generated from verified tasks, checked by the same tests |
4. The refinery, stage by stage
From raw pages to training shards
Each stage is cheaper than the next, so the expensive ones see the least data.
- Order matters for cost.
- Cheap rules run on everything; the model-based classifier runs only on what survives.
- In published pipelines such as FineWeb, the large majority of raw crawl is removed before training.
- Every stage writes what it removed and why.
- So a later surprise ("the model never learned X") can be traced back to a filter.
5. Filter 1: cheap rules
- The Gopher rules (DeepMind, 2021) are still the standard first pass.
- Document length, mean word length, symbol ratios, share of words with letters, presence of common English words, repeated lines.
- They are fast, explainable and catch spam, menus and broken extractions.
"""quality_filter.py โ heuristic document filters in the style of the
Gopher rules (Rae et al., 2021), the first gate of the data refinery.
Each rule is cheap and explainable; together they remove most junk before
any model-based classifier runs. SAMPLE documents are written for this
article. Thresholds follow the published Gopher rules where they exist.
"""
import re
SAMPLE = {
"tutorial": "To install the package, open a terminal and run the installer. "
"Then create a configuration file in your home directory and set the "
"path to your data. The tool reads the file on start-up and reports "
"any missing values, so check the log after the first run. " * 3,
"spam": "BUY NOW!!! $$$ best price best price best price click here click here "
"#### $$$ !!! ### deal deal deal $$$ ### !!! click click",
"boilerplate": "Home | About | Contact | Login\nHome | About | Contact | Login\n"
"Home | About | Contact | Login\nCopyright 2026. All rights reserved.",
"code_doc": "def area(r):\n return 3.14159 * r * r\n# compute the area of a circle "
"given its radius r; returns a float. Use it in geometry exercises "
"and check results against a calculator before submitting work. " * 2,
"short": "Click to continue.",
}
STOP = {"the", "be", "to", "of", "and", "that", "have", "with"}
def rules(doc):
words = re.findall(r"\S+", doc)
n = len(words)
alpha = [w for w in words if re.search(r"[A-Za-z]", w)]
lines = [l for l in doc.splitlines() if l.strip()]
return {
"50 <= words <= 100k": 50 <= n <= 100_000,
"mean word length 3-10": n > 0 and 3 <= sum(map(len, words)) / n <= 10,
"symbol/word ratio < 0.1": n > 0 and (doc.count("#") + doc.count("...")) / n < 0.1,
">= 80% words have a letter": n > 0 and len(alpha) / n >= 0.8,
">= 2 stop words": len({w.lower() for w in words} & STOP) >= 2,
"duplicate lines < 30%": not lines or 1 - len(set(lines)) / len(lines) < 0.30,
}
def main():
kept = 0
for name, doc in SAMPLE.items():
r = rules(doc)
failed = [k for k, ok in r.items() if not ok]
kept += not failed
print(f"{name:12s} {'KEEP' if not failed else 'DROP'} " + ("; ".join(failed) if failed else "passes all 6 rules"))
print(f"kept {kept} of {len(SAMPLE)}")
if __name__ == "__main__":
main()Five sample documents through six rules
Real output of quality_filter.py on documents written for this article.
- What the run shows.
- The tutorial and the documented code pass all six rules.
- Spam fails four rules, navigation boilerplate fails four, and a one-line page fails on length.
- What rules cannot do.
- They cannot tell a well-written wrong answer from a right one, or a dull page from an educational one.
- That is the job of the classifier in filter 3.
6. Filter 2: deduplication
- Duplicates waste compute and teach memorisation.
- The same press release on a thousand sites, mirrored docs, pages that differ only by a date.
- Exact hashing finds identical copies; near-duplicates need fuzzy matching.
- MinHash + locality-sensitive hashing is the workhorse.
- Estimate the overlap (Jaccard similarity) of each document's word 5-gram set from a short signature.
- Bucket signatures in bands so only likely pairs are compared, never all pairs.
"""minhash_dedup.py โ near-duplicate detection with MinHash + LSH bands.
Exact hashing misses pages that differ by a date or a menu; MinHash
estimates Jaccard similarity of word 5-gram sets cheaply, and banding
finds candidate pairs without comparing every pair. Documents are
written for this article. Standard library only; fixed seeds.
"""
import hashlib, itertools, random
DOCS = {
"a": "the cluster uses three power feeds per rack so that losing one feed leaves four of six supplies running in every node",
"b": "the cluster uses three power feeds per rack so that losing one feed leaves four of six supplies running in every node updated 2026",
"c": "posted today: the cluster uses three power feeds per rack so that losing one feed leaves four of six supplies running in every node",
"d": "gradient checkpointing trades extra compute in the backward pass for a large reduction in activation memory during training",
"e": "a mixture of experts routes each token to a few experts so that total parameters grow while compute per token stays fixed",
}
K, PERM, BANDS = 5, 128, 32 # 5-word shingles, 128 hashes, 32 bands of 4 rows
def shingles(t):
w = t.split()
return {" ".join(w[i:i + K]) for i in range(len(w) - K + 1)}
def h(s, seed):
return int.from_bytes(hashlib.blake2b(f"{seed}:{s}".encode(), digest_size=8).digest(), "big")
def signature(sh):
return [min(h(s, i) for s in sh) for i in range(PERM)]
def main():
sh = {k: shingles(v) for k, v in DOCS.items()}
sig = {k: signature(v) for k, v in sh.items()}
rows = PERM // BANDS
buckets = {}
for k, s in sig.items():
for b in range(BANDS):
buckets.setdefault((b, tuple(s[b * rows:(b + 1) * rows])), set()).add(k)
cand = {tuple(sorted(p)) for ks in buckets.values() if len(ks) > 1 for p in itertools.combinations(ks, 2)}
print(f"{'pair':5s} {'true Jaccard':>12s} {'MinHash est':>11s} candidate?")
for a, b in itertools.combinations(sorted(DOCS), 2):
j = len(sh[a] & sh[b]) / len(sh[a] | sh[b])
est = sum(x == y for x, y in zip(sig[a], sig[b])) / PERM
print(f"{a}-{b} {j:12.2f} {est:11.2f} {'yes' if (a, b) in cand else 'no'}")
keep, drop = [], set()
for a, b in sorted(cand):
if sum(x == y for x, y in zip(sig[a], sig[b])) / PERM >= 0.7:
drop.add(b)
print("drop as near-duplicates:", ", ".join(sorted(drop)) or "none")
# probability a pair with similarity s becomes a candidate: 1-(1-s^r)^b
for s in (0.5, 0.7, 0.8, 0.9):
print(f" P(candidate | similarity {s}) = {1 - (1 - s ** rows) ** BANDS:.3f}")
if __name__ == "__main__":
random.seed(0); main()Near-duplicate detection on five documents
Real output of minhash_dedup.py.
- Reading the output.
- Documents b and c are the same text as a with a date or a prefix added; MinHash estimates their overlap (0.89โ0.94) close to the true values (0.82โ0.90) and flags them.
- Unrelated documents score 0 and are never compared.
- The tuning knob is the banding.
- With 32 bands of 4 rows, any pair above about 0.7 similarity is almost certain to be checked; pairs at 0.5 are checked 87% of the time.
- More rows per band make the filter stricter; more bands make it catch more.
- A choice with consequences.
- FineWeb's authors reported that deduplicating each crawl separately worked better than deduplicating across all crawls at once, because global dedup removed too much good data.
- Gopala follows the published approach rather than inventing his own.
7. Filter 3: a quality classifier
- Train a small model to score pages by how much they teach.
- FineWeb-Edu used an LLM to rate a sample of pages for educational value, then trained a small classifier on those ratings to score the whole dataset.
- Gopala does the same with a twist: a second classifier scores engineering usefulness (code explanations, debugging write-ups, design docs), because his clients are engineering teams.
- Thresholds are chosen by ablation, not taste.
- Train small models on data kept at several thresholds and compare them (Part 9's ablations).
- Too strict a threshold removes diversity; too loose keeps junk.
- The classifier is itself a risk.
- It inherits the biases of whatever model labelled its training sample.
- He keeps 1% of rejected data in a random audit set and reads samples every week.
8. Planning the mixture
- The mixture is a set of shares that must add up to 10T tokens.
- Each source has a limited number of unique tokens after filtering.
- If the target share needs more tokens than exist, the source is repeated (more epochs).
- How much repetition is too much?
- Muennighoff et al. (2023) found that repeating data up to about four epochs costs little compared with fresh data, while much more repetition gives rapidly diminishing returns.
"""mixture.py โ plan Maestro-1's 10T-token mixture and check repetition.
A target share per source, the unique tokens available after filtering,
and how many times each source must be repeated (epochs). Repeating a
source a few times is mostly fine; many more starts to waste compute
(Muennighoff et al., 2023, "Scaling Data-Constrained Language Models").
Available-token figures are our planning assumptions, not measurements.
"""
TOTAL = 10e12
# source: (target share, unique tokens available after filtering)
SRC = {
"web (filtered, deduplicated)": (0.55, 9.0e12),
"code (permissive licences)": (0.20, 1.2e12),
"maths and science": (0.08, 0.35e12),
"books and long documents": (0.05, 0.25e12),
"multilingual web": (0.07, 1.5e12),
"verified skills (partners)": (0.02, 0.02e12),
"synthetic (from verified)": (0.03, 0.15e12),
}
def main():
assert abs(sum(s for s, _ in SRC.values()) - 1) < 1e-9
print(f"{'source':30s} {'share':>6s} {'tokens':>8s} {'unique':>8s} {'epochs':>7s}")
for name, (share, avail) in SRC.items():
t = share * TOTAL
ep = t / avail
flag = " <- heavy repetition" if ep > 4 else (" <- repeated" if ep > 1 else "")
print(f"{name:30s} {share:6.0%} {t/1e12:7.2f}T {avail/1e12:7.2f}T {ep:7.1f}{flag}")
uniq = sum(min(s * TOTAL, a) for s, a in SRC.values())
print(f"unique tokens seen: {uniq/1e12:.2f}T of {TOTAL/1e12:.0f}T ({uniq/TOTAL:.0%})")
if __name__ == "__main__":
main()- The finding that changed the plan.
- The verified skills from partners would be repeated 10 times at a 2% share: far past the point where repetition helps, and a recipe for memorising partner-derived examples.
- The fix he adopts.
- Lower verified skills to about 0.4% of the main blend (about two epochs), and use them mainly where they matter most: the final annealing phase at the end of pre-training and post-training (Part 11).
- Expand the synthetic share generated from verified tasks and checked by the same hidden tests, so the skill signal grows without repeating the same examples.
- Only about 82% of the 10T tokens are unique in the plan.
- That is normal for a start-up: it cannot crawl like a frontier lab, so code, maths and books are repeated around two times by design.
9. Try your own mixture
Mixture planner (our model)
10. How verified skills enter the data
From a partner task to training data
The same verifier that guards the loop (Part 1) guards the data.
- Abstracting is the confidentiality step.
- A skill core keeps the method ("how to bisect a flaky test") and drops the partner's identifiers, file names and business logic.
- A check scans every skill for strings that appear in the partner's repository; any hit blocks it.
- Three destinations, three different uses.
- The skill library for retrieval at inference time (Part 1's H1).
- A small share of annealing data at the end of pre-training (this part's mixture fix).
- RL tasks with hidden tests for post-training (Part 11).
11. Synthetic data, carefully
- Synthetic data can multiply a scarce signal.
- Generate variations of verified tasks (new inputs, renamed APIs, harder edge cases) and keep only those that pass their tests.
- It can also collapse diversity.
- Models trained heavily on their own outputs can lose rare knowledge over generations, the effect described as "model collapse" (Shumailov et al., 2024).
- His limits: synthetic data is generated only from verified seeds, always checked by tests, capped at a few percent of the blend, and labelled in the manifest.
- More on the trade-offs in the site's synthetic data article.
12. Records: what did the model see?
- Every shard has a manifest entry.
- Source and snapshot date, licence, filters and thresholds applied, dedup run id, token count, content hash.
- Why it matters later.
- Evaluation contamination checks (Part 12) need to know whether a benchmark leaked into training.
- Approvals (Part 14) and partners will ask what data trained the model.
- If a source has to be removed, the manifest says which future runs must exclude it.
- The pipeline itself is versioned.
- A sketch of the batch job, using a datatrove-style library, is below. It was written for this article and not run.
# pipeline.py โ the refinery as a batch job (sketch for a datatrove-style
# library; class names follow Hugging Face datatrove's documented blocks,
# but check them against the version you install).
from datatrove.executor import SlurmPipelineExecutor
from datatrove.pipeline.readers import WarcReader
from datatrove.pipeline.extractors import Trafilatura
from datatrove.pipeline.filters import LanguageFilter, GopherQualityFilter, GopherRepetitionFilter, URLFilter
from datatrove.pipeline.writers import JsonlWriter
SlurmPipelineExecutor(
job_name="refinery-cc-2026-10",
pipeline=[
WarcReader("s3://maestro-raw/cc/2026-10/", glob_pattern="*.warc.gz"),
URLFilter(), # blocklists: adult, malware, opted-out domains
Trafilatura(favour_precision=True), # HTML -> main text
LanguageFilter(languages=["en"]),
GopherRepetitionFilter(),
GopherQualityFilter(), # the rules in quality_filter.py
JsonlWriter("s3://maestro-clean/cc/2026-10/"),
],
tasks=2000, time="12:00:00", partition="exp", # CPU-heavy: runs on the pre-emptible partition
logging_dir="s3://maestro-logs/refinery/cc-2026-10/",
).run()13. What the refinery costs
- Mostly CPU, storage and time.
- Extraction, rules and dedup are CPU work; the classifier needs some GPU time.
- It runs on the pre-emptible partition and spare CPUs, so its GPU cost is mostly absorbed by idle hours in the main block (Part 4's capacity plan).
- The bigger cost is engineering attention.
- Weeks of reading samples, tuning thresholds and running ablations. Gopala delegates the reading to agents with a strict rubric and checks a random 10% himself.
- In the story, the refinery is ready in week 26 and the frozen blend for the main run in week 36, two weeks later than planned because the mixture fix needed a second round of ablations.
- These dates are the story, not measurements.
14. Personal data and harmful content
- Web data contains people.
- Email addresses, phone numbers, home addresses, names attached to private details.
- He removes or masks common personal-data patterns with rules and a named-entity model, and drops pages from domains that exist to expose people.
- Harmful content filters run before training, not only after.
- Classifiers for sexual content involving minors (zero tolerance, hash-matching against known material through established services), extreme violence and hate.
- The aim is not a sanitised model; it is not teaching it material no customer should ever receive.
- Every removal is counted per source.
- If a filter removes far more from one language or region than others, it is checked for bias before it stays.
15. Keeping benchmarks out of the training data
- If test questions leak into training, scores mean nothing.
- Benchmarks are on the web; a crawl will contain some of them.
- The GPT-3 paper (Brown et al., 2020) checked for overlapping 13-word sequences between training data and benchmarks; n-gram overlap checks of this kind are still the standard first pass.
- His procedure.
- Before freezing the blend, scan every shard for long n-gram overlaps with every benchmark in the evaluation plan (Part 12) and with the partners' held-out tasks.
- Remove matches; record what was removed in the manifest.
- Keep a private, never-published task set that cannot have leaked.
- The Evaluation Layer article on contamination covers why this is never fully solved.
16. A week in the refinery
| Day | Work | Who |
|---|---|---|
| Monday | New crawl snapshot and partner skills arrive; pipeline runs overnight | Automated |
| Tuesday | Read 200 random kept and 200 random rejected documents per source | Agents with a rubric; Gopala checks 10% |
| Wednesday | Adjust a threshold if the samples show a problem; re-run affected shards | Gopala |
| Thursday | Small ablation on any changed source | Automated on the exp partition |
| Friday | Update the manifest and the source register | Gopala |
17. Eight data mistakes
- Training on everything downloadable and asking about rights later.
- Deduplicating so aggressively that most good data disappears.
- Trusting a quality classifier without reading what it rejects.
- Repeating a small precious dataset many times at full share.
- Letting partner code leak into pre-training data.
- Using synthetic data without a verifier in the loop.
- Keeping no manifest, so contamination cannot be checked later.
- Changing the blend mid-run without recording it.
18. Delegation box
- Source register and rights decisions
- Mixture choice and the verified-skill fix
- Freezing the main-run blend
- Reading and grading rejected-data samples against a rubric
- Licence detection triage
- Writing manifest summaries
- Public filtered datasets as a base
- Open-source data tooling
- A data-licensing lawyer for the register
19. Budget and risk tracker
| Item | Status after Part 8 | Note |
|---|---|---|
| Spend to date | ~$1.9M (scenario) | Mostly compute for Maestro-0 and ablations; refinery runs in idle hours |
| Data | 10T-token blend frozen (scenario) | ~82% unique; verified skills capped and moved to annealing |
| Time | Week 36 (month ~8.5) | Two weeks late (scenario) |
| Data-rights risk | Managed | Register, opt-outs, no partner code |
| Contamination risk | Managed | Manifest per shard for Part 12 |
| Skill-signal risk | Lower | Mixture fix found by the script |
20. The series
The Maestro: 18 parts in five phases
- The refinery turns 10 trillion tokens of raw material into a blend Gopala can defend: filtered, deduplicated, scored, rights-checked and recorded.
- The script found the plan's biggest flaw (repeating partner skills ten times) before any GPU paid for it.
- Next, Part 9: the model that will eat this data, and how to spread it across 1,024 GPUs.
21. Reading the numbers
- Primary / standard
- FineWeb ~15T tokens; FineWeb-Edu 1.3T tokens; per-crawl dedup finding (Hugging Face). Llama 3: ~15T tokens; mix ~50% general, 25% maths and reasoning, 17% code, 8% multilingual (Meta). Gopher quality rules (Rae et al., 2021). Repetition up to ~4 epochs (Muennighoff et al., 2023). Model collapse (Shumailov et al., Nature 2024). MinHash (Broder, 1997).
- Secondary โ ๏ธ
- None in this part.
- Our assumptions
- All available-token counts, shares, the mixture fix, sample documents, dates and costs.