Home โ€บ Blog โ€บ The Maestro ยท Part 8
The Maestro ยท Part 8 ยท Phase 2 ยท ๐Ÿง  ML engineer hat

The Data Refinery

Phase 2 starts with what the machine eats. Maestro-1 needs about 10 trillion tokens (Part 3), and the quality of those tokens matters as much as the size of the model. This part builds the data refinery that turns raw web pages, code and partner tasks into a training mixture: where the data comes from and what Gopala may use, three filters with runnable code, deduplication, the mixture plan, how verified skills enter the data, and the records that let him answer "what did the model see?" a year later.

FL
FrontierAGI Team
๐Ÿง  ML engineer๐Ÿ—ƒ๏ธ Data engineerโš–๏ธ Data-rights owner๐Ÿ’ป Coder
โš ๏ธ How to read this (as of 30 Oct 2026) Dataset sizes (FineWeb, FineWeb-Edu), Llama 3's data mix and the Gopher rules are from their published papers and cards. quality_filter.py, minhash_dedup.py and mixture.py were run (real output below) on documents and token counts written for this article; available-token figures are our planning assumptions. pipeline.py uses an external library and was not executed for this article. Gopala is a teaching scenario. Disclosure: Claude, made by Anthropic, helps write this site.

1. Mission log: weeks 14โ€“36

Weeks 14โ€“18 ยท Sources and rights
  • Lists every candidate source with its licence, terms and opt-out status; drops anything he cannot defend to a partner's lawyer.
  • Downloads public filtered web datasets as a starting point instead of crawling everything himself.
Weeks 18โ€“26 ยท The refinery
  • Builds the filter โ†’ dedup โ†’ classify pipeline on the pre-emptible partition (Part 7), using spare GPU and CPU hours between Maestro-0 jobs.
  • Writes the three scripts in this part and runs them on samples by hand before trusting them at scale.
Weeks 26โ€“36 ยท Mixture and records
  • Runs small "data ablations" (Part 9) to choose the mixture; freezes the 10T-token blend for the main run.
  • Every shard gets a manifest entry: source, licence, filters applied, hash.

2. Why data decides the model

Where 10 trillion tokens come from (planned shares)

Web, filtered and deduplicated55%
Code, permissive licences20%
Maths and science8%
Multilingual web7%
Books and long documents5%
Synthetic, from verified skills3%
Verified skills from partners2%

Our planned mixture for Maestro-1 (see mixture.py below). Llama 3's published mix is shown in the text for comparison.

3. What he may use

SourceUse?Why
Public filtered web datasets (e.g. FineWeb)Yes, as a baseDocumented pipeline, published licence terms, opt-outs honoured upstream
Own crawl of selected domainsYes, limitedRespect robots.txt and opt-outs; identify the crawler
Permissively licensed codeYesLicence detected per file; copyleft excluded
Paywalled news and books without a licenceNoLegal risk outweighs value
Partner repositoriesNoOnly verified skills derived from tasks, under contract
Synthetic data from his own modelsYesGenerated from verified tasks, checked by the same tests

4. The refinery, stage by stage

From raw pages to training shards

๐ŸŒExtractHTML โ†’ main text; drop menus and ads
๐ŸŒLanguageKeep target languages
๐ŸงนHeuristicsCheap rules (Gopher-style)
๐Ÿ”DedupExact, then MinHash near-duplicates
๐ŸŽ“ClassifyModel-based quality score
๐Ÿ›ก๏ธSafety & PIIRemove personal data, toxic content
๐Ÿ“ฆShardTokenise, record manifest

Each stage is cheaper than the next, so the expensive ones see the least data.

5. Filter 1: cheap rules

quality_filter.py ยท Python 3, no dependencies
"""quality_filter.py โ€” heuristic document filters in the style of the
Gopher rules (Rae et al., 2021), the first gate of the data refinery.

Each rule is cheap and explainable; together they remove most junk before
any model-based classifier runs. SAMPLE documents are written for this
article. Thresholds follow the published Gopher rules where they exist.
"""
import re

SAMPLE = {
    "tutorial": "To install the package, open a terminal and run the installer. "
                "Then create a configuration file in your home directory and set the "
                "path to your data. The tool reads the file on start-up and reports "
                "any missing values, so check the log after the first run. " * 3,
    "spam":     "BUY NOW!!! $$$ best price best price best price click here click here "
                "#### $$$ !!! ### deal deal deal $$$ ### !!! click click",
    "boilerplate": "Home | About | Contact | Login\nHome | About | Contact | Login\n"
                   "Home | About | Contact | Login\nCopyright 2026. All rights reserved.",
    "code_doc": "def area(r):\n    return 3.14159 * r * r\n# compute the area of a circle "
                "given its radius r; returns a float. Use it in geometry exercises "
                "and check results against a calculator before submitting work. " * 2,
    "short":    "Click to continue.",
}
STOP = {"the", "be", "to", "of", "and", "that", "have", "with"}

def rules(doc):
    words = re.findall(r"\S+", doc)
    n = len(words)
    alpha = [w for w in words if re.search(r"[A-Za-z]", w)]
    lines = [l for l in doc.splitlines() if l.strip()]
    return {
        "50 <= words <= 100k": 50 <= n <= 100_000,
        "mean word length 3-10": n > 0 and 3 <= sum(map(len, words)) / n <= 10,
        "symbol/word ratio < 0.1": n > 0 and (doc.count("#") + doc.count("...")) / n < 0.1,
        ">= 80% words have a letter": n > 0 and len(alpha) / n >= 0.8,
        ">= 2 stop words": len({w.lower() for w in words} & STOP) >= 2,
        "duplicate lines < 30%": not lines or 1 - len(set(lines)) / len(lines) < 0.30,
    }

def main():
    kept = 0
    for name, doc in SAMPLE.items():
        r = rules(doc)
        failed = [k for k, ok in r.items() if not ok]
        kept += not failed
        print(f"{name:12s} {'KEEP' if not failed else 'DROP'}  " + ("; ".join(failed) if failed else "passes all 6 rules"))
    print(f"kept {kept} of {len(SAMPLE)}")

if __name__ == "__main__":
    main()
$ python3 quality_filter.py # on five sample documents written for this article tutorial KEEP passes all 6 rules spam DROP 50 <= words <= 100k; symbol/word ratio < 0.1; >= 80% words have a letter; >= 2 stop words boilerplate DROP 50 <= words <= 100k; >= 80% words have a letter; >= 2 stop words; duplicate lines < 30% code_doc KEEP passes all 6 rules short DROP 50 <= words <= 100k; >= 2 stop words kept 2 of 5

Five sample documents through six rules

Tutorialpasses 6/6 ยท KEEP
Documented codepasses 6/6 ยท KEEP
One-line pagepasses 4/6 ยท DROP
Spampasses 2/6 ยท DROP
Navigation boilerplatepasses 2/6 ยท DROP

Real output of quality_filter.py on documents written for this article.

6. Filter 2: deduplication

minhash_dedup.py ยท Python 3, no dependencies
"""minhash_dedup.py โ€” near-duplicate detection with MinHash + LSH bands.

Exact hashing misses pages that differ by a date or a menu; MinHash
estimates Jaccard similarity of word 5-gram sets cheaply, and banding
finds candidate pairs without comparing every pair. Documents are
written for this article. Standard library only; fixed seeds.
"""
import hashlib, itertools, random

DOCS = {
    "a": "the cluster uses three power feeds per rack so that losing one feed leaves four of six supplies running in every node",
    "b": "the cluster uses three power feeds per rack so that losing one feed leaves four of six supplies running in every node updated 2026",
    "c": "posted today: the cluster uses three power feeds per rack so that losing one feed leaves four of six supplies running in every node",
    "d": "gradient checkpointing trades extra compute in the backward pass for a large reduction in activation memory during training",
    "e": "a mixture of experts routes each token to a few experts so that total parameters grow while compute per token stays fixed",
}
K, PERM, BANDS = 5, 128, 32          # 5-word shingles, 128 hashes, 32 bands of 4 rows

def shingles(t):
    w = t.split()
    return {" ".join(w[i:i + K]) for i in range(len(w) - K + 1)}

def h(s, seed):
    return int.from_bytes(hashlib.blake2b(f"{seed}:{s}".encode(), digest_size=8).digest(), "big")

def signature(sh):
    return [min(h(s, i) for s in sh) for i in range(PERM)]

def main():
    sh = {k: shingles(v) for k, v in DOCS.items()}
    sig = {k: signature(v) for k, v in sh.items()}
    rows = PERM // BANDS
    buckets = {}
    for k, s in sig.items():
        for b in range(BANDS):
            buckets.setdefault((b, tuple(s[b * rows:(b + 1) * rows])), set()).add(k)
    cand = {tuple(sorted(p)) for ks in buckets.values() if len(ks) > 1 for p in itertools.combinations(ks, 2)}
    print(f"{'pair':5s} {'true Jaccard':>12s} {'MinHash est':>11s}  candidate?")
    for a, b in itertools.combinations(sorted(DOCS), 2):
        j = len(sh[a] & sh[b]) / len(sh[a] | sh[b])
        est = sum(x == y for x, y in zip(sig[a], sig[b])) / PERM
        print(f"{a}-{b}   {j:12.2f} {est:11.2f}  {'yes' if (a, b) in cand else 'no'}")
    keep, drop = [], set()
    for a, b in sorted(cand):
        if sum(x == y for x, y in zip(sig[a], sig[b])) / PERM >= 0.7:
            drop.add(b)
    print("drop as near-duplicates:", ", ".join(sorted(drop)) or "none")
    # probability a pair with similarity s becomes a candidate: 1-(1-s^r)^b
    for s in (0.5, 0.7, 0.8, 0.9):
        print(f"  P(candidate | similarity {s}) = {1 - (1 - s ** rows) ** BANDS:.3f}")

if __name__ == "__main__":
    random.seed(0); main()
$ python3 minhash_dedup.py pair true Jaccard MinHash est candidate? a-b 0.90 0.93 yes a-c 0.90 0.94 yes a-d 0.00 0.00 no a-e 0.00 0.00 no b-c 0.82 0.89 yes b-d 0.00 0.00 no b-e 0.00 0.00 no c-d 0.00 0.00 no c-e 0.00 0.00 no d-e 0.00 0.00 no drop as near-duplicates: b, c P(candidate | similarity 0.5) = 0.873 P(candidate | similarity 0.7) = 1.000 P(candidate | similarity 0.8) = 1.000 P(candidate | similarity 0.9) = 1.000

Near-duplicate detection on five documents

0.93 / 0.94MinHash estimate for the two edited copies of a
0.90true overlap for both
2 of 5dropped as near-duplicates
0 of 7unrelated pairs compared

Real output of minhash_dedup.py.

7. Filter 3: a quality classifier

8. Planning the mixture

mixture.py ยท Python 3, no dependencies
"""mixture.py โ€” plan Maestro-1's 10T-token mixture and check repetition.

A target share per source, the unique tokens available after filtering,
and how many times each source must be repeated (epochs). Repeating a
source a few times is mostly fine; many more starts to waste compute
(Muennighoff et al., 2023, "Scaling Data-Constrained Language Models").
Available-token figures are our planning assumptions, not measurements.
"""
TOTAL = 10e12
# source: (target share, unique tokens available after filtering)
SRC = {
    "web (filtered, deduplicated)": (0.55, 9.0e12),
    "code (permissive licences)":   (0.20, 1.2e12),
    "maths and science":            (0.08, 0.35e12),
    "books and long documents":     (0.05, 0.25e12),
    "multilingual web":             (0.07, 1.5e12),
    "verified skills (partners)":   (0.02, 0.02e12),
    "synthetic (from verified)":    (0.03, 0.15e12),
}

def main():
    assert abs(sum(s for s, _ in SRC.values()) - 1) < 1e-9
    print(f"{'source':30s} {'share':>6s} {'tokens':>8s} {'unique':>8s} {'epochs':>7s}")
    for name, (share, avail) in SRC.items():
        t = share * TOTAL
        ep = t / avail
        flag = "  <- heavy repetition" if ep > 4 else ("  <- repeated" if ep > 1 else "")
        print(f"{name:30s} {share:6.0%} {t/1e12:7.2f}T {avail/1e12:7.2f}T {ep:7.1f}{flag}")
    uniq = sum(min(s * TOTAL, a) for s, a in SRC.values())
    print(f"unique tokens seen: {uniq/1e12:.2f}T of {TOTAL/1e12:.0f}T ({uniq/TOTAL:.0%})")

if __name__ == "__main__":
    main()
$ python3 mixture.py source share tokens unique epochs web (filtered, deduplicated) 55% 5.50T 9.00T 0.6 code (permissive licences) 20% 2.00T 1.20T 1.7 <- repeated maths and science 8% 0.80T 0.35T 2.3 <- repeated books and long documents 5% 0.50T 0.25T 2.0 <- repeated multilingual web 7% 0.70T 1.50T 0.5 verified skills (partners) 2% 0.20T 0.02T 10.0 <- heavy repetition synthetic (from verified) 3% 0.30T 0.15T 2.0 <- repeated unique tokens seen: 8.17T of 10T (82%)

9. Try your own mixture

Mixture planner (our model)

10. How verified skills enter the data

From a partner task to training data

๐ŸงฉTaskReal engineering task from a partner, under contract
๐Ÿค–AttemptMaestro-0 solves it in a sandbox
โœ…VerifyHidden tests pass; no regression
โœ‚๏ธAbstractStrip partner code; keep the reusable skill core
๐Ÿ”’ReviewLicence and confidentiality check
๐Ÿ“ฆDataSkill library + annealing data + RL tasks

The same verifier that guards the loop (Part 1) guards the data.

11. Synthetic data, carefully

12. Records: what did the model see?

pipeline.py ยท datatrove-style pipeline sketch ยท not executed for this article
# pipeline.py โ€” the refinery as a batch job (sketch for a datatrove-style
# library; class names follow Hugging Face datatrove's documented blocks,
# but check them against the version you install).
from datatrove.executor import SlurmPipelineExecutor
from datatrove.pipeline.readers import WarcReader
from datatrove.pipeline.extractors import Trafilatura
from datatrove.pipeline.filters import LanguageFilter, GopherQualityFilter, GopherRepetitionFilter, URLFilter
from datatrove.pipeline.writers import JsonlWriter

SlurmPipelineExecutor(
    job_name="refinery-cc-2026-10",
    pipeline=[
        WarcReader("s3://maestro-raw/cc/2026-10/", glob_pattern="*.warc.gz"),
        URLFilter(),                       # blocklists: adult, malware, opted-out domains
        Trafilatura(favour_precision=True),  # HTML -> main text
        LanguageFilter(languages=["en"]),
        GopherRepetitionFilter(),
        GopherQualityFilter(),             # the rules in quality_filter.py
        JsonlWriter("s3://maestro-clean/cc/2026-10/"),
    ],
    tasks=2000, time="12:00:00", partition="exp",   # CPU-heavy: runs on the pre-emptible partition
    logging_dir="s3://maestro-logs/refinery/cc-2026-10/",
).run()

13. What the refinery costs

Scenario
  • In the story, the refinery is ready in week 26 and the frozen blend for the main run in week 36, two weeks later than planned because the mixture fix needed a second round of ablations.
  • These dates are the story, not measurements.

14. Personal data and harmful content

15. Keeping benchmarks out of the training data

16. A week in the refinery

DayWorkWho
MondayNew crawl snapshot and partner skills arrive; pipeline runs overnightAutomated
TuesdayRead 200 random kept and 200 random rejected documents per sourceAgents with a rubric; Gopala checks 10%
WednesdayAdjust a threshold if the samples show a problem; re-run affected shardsGopala
ThursdaySmall ablation on any changed sourceAutomated on the exp partition
FridayUpdate the manifest and the source registerGopala

17. Eight data mistakes

18. Delegation box

๐Ÿ™‹ Gopala himself
  • Source register and rights decisions
  • Mixture choice and the verified-skill fix
  • Freezing the main-run blend
๐Ÿค– AI agents
  • Reading and grading rejected-data samples against a rubric
  • Licence detection triage
  • Writing manifest summaries
๐Ÿค Bought, rented or borrowed
  • Public filtered datasets as a base
  • Open-source data tooling
  • A data-licensing lawyer for the register

19. Budget and risk tracker

ItemStatus after Part 8Note
Spend to date~$1.9M (scenario)Mostly compute for Maestro-0 and ablations; refinery runs in idle hours
Data10T-token blend frozen (scenario)~82% unique; verified skills capped and moved to annealing
TimeWeek 36 (month ~8.5)Two weeks late (scenario)
Data-rights riskManagedRegister, opt-outs, no partner code
Contamination riskManagedManifest per shard for Part 12
Skill-signal riskLowerMixture fix found by the script

20. The series

The Maestro: 18 parts in five phases

Verdict
  • The refinery turns 10 trillion tokens of raw material into a blend Gopala can defend: filtered, deduplicated, scored, rights-checked and recorded.
    • The script found the plan's biggest flaw (repeating partner skills ten times) before any GPU paid for it.
    • Next, Part 9: the model that will eat this data, and how to spread it across 1,024 GPUs.

21. Reading the numbers

Primary / standard
FineWeb ~15T tokens; FineWeb-Edu 1.3T tokens; per-crawl dedup finding (Hugging Face). Llama 3: ~15T tokens; mix ~50% general, 25% maths and reasoning, 17% code, 8% multilingual (Meta). Gopher quality rules (Rae et al., 2021). Repetition up to ~4 epochs (Muennighoff et al., 2023). Model collapse (Shumailov et al., Nature 2024). MinHash (Broder, 1997).
Secondary โš ๏ธ
None in this part.
Our assumptions
All available-token counts, shares, the mixture fix, sample documents, dates and costs.