Home โ€บ Blog โ€บ The Maestro, Season 2 ยท Part 13
The Maestro, Season 2 ยท Part 13 ยท Phase 7 ยท ๐Ÿงช Research director hat

New Ingredients

Every new model is a recipe: an architecture, a data mixture and a training method, and a long queue of ideas that want a place in it. This part follows the two weeks in which Maestro-2's research team tested seven candidates at small scale, threw out three, and found the one change that would save five million dollars without making the model any smarter.

FL
FrontierAGI Team
๐Ÿ”ฌ Research director๐Ÿ‹๏ธ Pre-training๐Ÿ’ฐ Finance
โš ๏ธ How to read this (as of 4 Nov 2026) Gopala's company is a teaching scenario. ablation_ladder.py and fp8_budget.py were run (real output below). The ingredients are real published ideas (sources at the end); the measured gains, seeds and achieved FP8 speed-ups are scenario numbers. Disclosure: Claude, made by Anthropic, helps write this site.

1. Month 37: the queue of ideas

๐Ÿ“– The story

By the second week of month 37 the research wiki had a page called Maestro-2: please include. It had forty-one entries.

Mira cut it to seven over a weekend, using one rule: anything without a published result at more than a billion parameters, or a clear story for why it would help the 21% of failures, was out. Then she gave the seven to the small-scale team with a warning written at the top of the page. We are not looking for good ideas. We are looking for ideas that survive being made bigger.

๐Ÿ”ฌ In plain words: ablation An ablation is a controlled experiment: train two models that are identical except for one change, and measure the difference. Doing it at small scale is cheap; the danger is that some changes help small models and do nothing for large ones. So each idea is tested at two sizes, several times, before it can enter a run that costs millions.

2. The map: seven ideas through one test

๐Ÿ•ธ๏ธ Which ingredients survive the two-scale test

or click any node

Green ideas passed, red ideas failed (red arrows), orange ideas passed a different test: no worse, but cheaper.

3. Seven candidates, two sizes, three seeds

ablation_ladder.py ยท Python 3, no dependencies; real ideas, scenario measurements
"""ablation_ladder.py โ€” which new ingredients earn a place in Maestro-2?

Each candidate is trained at two small scales (1B and 3B active) with three
seeds each, against a baseline. An ingredient is kept only if the mean gain is
more than twice its standard error at BOTH scales (small effects that vanish
with scale are the classic trap). Measured gains (points on the held-out task
suite) are SCENARIO numbers; the ideas are real (see the article's sources).
"""
import statistics as st
RUNS = {  # ingredient: {scale: [gain per seed]}
    "multi-token prediction": {"1B": [0.6, 0.9, 0.7], "3B": [0.8, 1.0, 0.9]},
    "aux-loss-free load balancing": {"1B": [0.3, 0.5, 0.4], "3B": [0.4, 0.6, 0.5]},
    "FP8 training (matched loss)": {"1B": [0.0, 0.1, -0.1], "3B": [0.1, 0.0, 0.0]},
    "environment data in mid-training": {"1B": [1.1, 0.7, 1.4], "3B": [1.5, 1.2, 1.8]},
    "+10% code in the mixture": {"1B": [0.9, 0.2, 1.3], "3B": [0.1, -0.4, 0.5]},
    "new 200k tokenizer": {"1B": [0.5, -0.6, 0.9], "3B": [0.3, -0.2, 0.2]},
    "wider attention heads": {"1B": [0.4, 0.3, 0.5], "3B": [-0.1, 0.0, 0.1]}}

NONINF = {"FP8 training (matched loss)"}   # judged on "no worse" (mean > -0.2): its prize is speed

def main():
    keep = []
    for name, sc in RUNS.items():
        cells, ok = [], True
        for s in ("1B", "3B"):
            m, se = st.mean(sc[s]), st.stdev(sc[s]) / len(sc[s]) ** 0.5
            ok &= (m > -0.2) if name in NONINF else (m > 2 * se); cells.append(f"{s} {m:+.2f}ยฑ{se:.2f}")
        verdict = ("KEEP (speed)" if name in NONINF else "KEEP") if ok else "drop"
        if ok: keep.append(name)
        print(f"{name:34} {'   '.join(cells)}   {verdict}")
    print(f"\nkept {len(keep)} of {len(RUNS)}: {', '.join(keep)}")
    print("FP8 passes a no-worse test, not a gain test: its prize is speed (see fp8_budget.py)")

if __name__ == "__main__":
    main()
$ python3 ablation_ladder.py # keep only what survives both scales multi-token prediction 1B +0.73ยฑ0.09 3B +0.90ยฑ0.06 KEEP aux-loss-free load balancing 1B +0.40ยฑ0.06 3B +0.50ยฑ0.06 KEEP FP8 training (matched loss) 1B +0.00ยฑ0.06 3B +0.03ยฑ0.03 KEEP (speed) environment data in mid-training 1B +1.07ยฑ0.20 3B +1.50ยฑ0.17 KEEP +10% code in the mixture 1B +0.80ยฑ0.32 3B +0.07ยฑ0.26 drop new 200k tokenizer 1B +0.27ยฑ0.45 3B +0.10ยฑ0.15 drop wider attention heads 1B +0.40ยฑ0.06 3B +0.00ยฑ0.06 drop kept 4 of 7: multi-token prediction, aux-loss-free load balancing, FP8 training (matched loss), environment data in mid-training FP8 passes a no-worse test, not a gain test: its prize is speed (see fp8_budget.py)

Gain on the held-out suite at 1B โ†’ 3B active parameters

Environment data in mid-training+1.07 โ†’ +1.50
Multi-token prediction+0.73 โ†’ +0.90
Aux-loss-free balancing+0.40 โ†’ +0.50
+10% code in the mixture+0.80 โ†’ +0.07
New 200k tokenizer+0.27 โ†’ +0.10 (noisy)
Wider attention heads+0.40 โ†’ 0.00

Real output, bar = 3B gain. Three of the four kept ideas grow with scale; the three dropped ones shrink or are noise.

๐Ÿ“– The story

The +10% code result was the one that hurt. At 1B it was the second-best idea on the board, and two engineers had spent a month building the mixture. At 3B it was nothing.

This is why we test twice, Mira said in the review. Small models are short of everything. Give them more of anything useful and they improve. Bigger models already have enough code. The test is not whether an idea helps. It is whether it still helps when the model no longer needs help.

๐Ÿ”Ž Go deeper: what each kept idea does
  • Multi-token prediction: an extra output head learns to predict two tokens ahead.
    • It gives the model a richer training signal per token and can also be used for faster decoding.
  • Aux-loss-free load balancing: a per-expert bias steers tokens away from overloaded experts.
    • Unlike the usual extra loss term, it does not pull the model away from its main objective.
  • Environment data: traces of agents solving the Part 8 environments, mixed in late in pre-training.
    • The company's own advantage: no rival has these environments.

4. The ingredient that only saves money

๐Ÿ”ฌ In plain words: FP8 Numbers inside a model are stored with a fixed number of bits. Training usually uses 16-bit numbers (BF16). FP8 uses 8 bits for the biggest calculations, so a GPU can do about twice as many per second. The risk is precision: too few bits and training becomes unstable. The trick, shown at scale in DeepSeek-V3, is to use FP8 only where it is safe and keep sensitive parts in higher precision.
fp8_budget.py ยท Python 3, no dependencies; NVIDIA peak figures, scenario speed-ups
"""fp8_budget.py โ€” what FP8 training buys Maestro-2.

H100 peak dense throughput doubles from BF16 (989 TFLOP/s) to FP8 (1,979).
Real end-to-end gains are far smaller: not every operation runs in FP8, and
communication and memory traffic do not speed up. The achieved speed-ups are
SCENARIO assumptions bracketing what public reports describe.
"""
FLOPS, PEAK, MFU, PRICE, GPUS = 1.06e25, 989e12, 0.40, 2.40, 4096

def main():
    base = FLOPS / (PEAK * MFU * 0.91) / 3600
    print(f"{'precision':26} {'GPU-hours':>10} {'days on 4,096':>14} {'cost $M':>8}")
    for name, sp in (("BF16 (baseline)", 1.0), ("FP8, cautious (x1.25)", 1.25), ("FP8, planned (x1.35)", 1.35), ("FP8, ideal peak (x2.0)", 2.0)):
        gh = base / sp
        print(f"{name:26} {gh / 1e6:9.2f}M {gh / GPUS / 24:14.0f} {gh * PRICE / 1e6:8.1f}")
    print(f"\nplanned saving: {(base - base / 1.35) * PRICE / 1e6:.1f} $M and {(base - base / 1.35) / GPUS / 24:.0f} days")

if __name__ == "__main__":
    main()
$ python3 fp8_budget.py # what FP8 buys on 4,096 GPUs precision GPU-hours days on 4,096 cost $M BF16 (baseline) 8.18M 83 19.6 FP8, cautious (x1.25) 6.54M 67 15.7 FP8, planned (x1.35) 6.06M 62 14.5 FP8, ideal peak (x2.0) 4.09M 42 9.8 planned saving: 5.1 $M and 22 days

Days on 4,096 GPUs, and cost

BF16 baseline ยท $19.6M83 days
FP8 cautious ร—1.25 ยท $15.7M67 days
FP8 planned ร—1.35 ยท $14.5M62 days
FP8 at ideal peak ร—2.0 ยท $9.8M42 days (never reached)

Real output. The planned speed-up is 1.35ร—, far below the 2ร— on the data sheet, because communication and memory traffic do not get faster.

๐Ÿ“– The story

Arjun had been sceptical. He had seen FP8 runs diverge. So the team did one more thing before committing: a 3B run, matched step for step against BF16, for twice as long as the ablations. The two loss curves sat on top of each other to the third decimal place.

Five million dollars, said the finance lead, for a change nobody will ever see. That is the best kind, said Gopala.

5. Sandbox: pick the recipe

๐ŸŽฎ Sandbox: pick the recipe

Gains are the 3B-scale means from ablation_ladder.py; days and cost from fp8_budget.py. Adding gains is a simplification: real ingredients interact, which is why the kept set is re-tested together before the run.

6. Five mistakes

7. Delegation box

๐Ÿ™‹ Gopala himself
  • The final recipe and the kill list
๐Ÿค– AI agents
  • Running 42 small training runs
  • Comparing loss curves
๐Ÿค Bought, rented or borrowed
  • Spare capacity on the 4,096-GPU block before the main run

8. Budget and risk tracker

ItemStatus after Part 13Note
Recipe4 of 7 keptMTP, balancing, environment data, FP8
Expected gain+2.90 points at 3Bsum of kept gains, re-tested together
Compute$14.5M plannedFP8 saves $5.1M and 22 days
FP8 riskWatcha matched 3B run showed no loss gap

9. The series

The Maestro, Season 2: 23 parts in five phases (Season 1: 18 parts)

VerdictMaestro-2's recipe keeps four of seven candidates: three that still help at larger scale and FP8, which makes no one smarter but cuts the run from 83 to 62 days and about $5M. Ideas that only help small models were dropped, however good they looked. Next: the run.

10. Reading the numbers

Primary / standard
DeepSeek-V3 technical report (multi-token prediction, auxiliary-loss-free balancing, FP8 training at scale); Gloeckle et al. 2024 on multi-token prediction; Micikevicius et al. 2022 on FP8 formats; NVIDIA H100 peak figures.
Secondary โš ๏ธ
None in this part.
Our assumptions
Measured gains, seeds, achieved speed-ups and all story events are the scenario.