New Ingredients
Every new model is a recipe: an architecture, a data mixture and a training method, and a long queue of ideas that want a place in it. This part follows the two weeks in which Maestro-2's research team tested seven candidates at small scale, threw out three, and found the one change that would save five million dollars without making the model any smarter.
ablation_ladder.py and fp8_budget.py were run (real output below). The ingredients are real published ideas (sources at the end); the measured gains, seeds and achieved FP8 speed-ups are scenario numbers. Disclosure: Claude, made by Anthropic, helps write this site.
1. Month 37: the queue of ideas
By the second week of month 37 the research wiki had a page called Maestro-2: please include. It had forty-one entries.
Mira cut it to seven over a weekend, using one rule: anything without a published result at more than a billion parameters, or a clear story for why it would help the 21% of failures, was out. Then she gave the seven to the small-scale team with a warning written at the top of the page. We are not looking for good ideas. We are looking for ideas that survive being made bigger.
2. The map: seven ideas through one test
๐ธ๏ธ Which ingredients survive the two-scale test
Green ideas passed, red ideas failed (red arrows), orange ideas passed a different test: no worse, but cheaper.
- One test for everyone.
- Three seeds at 1B and three at 3B active parameters.
- Keep only if the mean gain is more than twice its standard error at both sizes.
3. Seven candidates, two sizes, three seeds
"""ablation_ladder.py โ which new ingredients earn a place in Maestro-2?
Each candidate is trained at two small scales (1B and 3B active) with three
seeds each, against a baseline. An ingredient is kept only if the mean gain is
more than twice its standard error at BOTH scales (small effects that vanish
with scale are the classic trap). Measured gains (points on the held-out task
suite) are SCENARIO numbers; the ideas are real (see the article's sources).
"""
import statistics as st
RUNS = { # ingredient: {scale: [gain per seed]}
"multi-token prediction": {"1B": [0.6, 0.9, 0.7], "3B": [0.8, 1.0, 0.9]},
"aux-loss-free load balancing": {"1B": [0.3, 0.5, 0.4], "3B": [0.4, 0.6, 0.5]},
"FP8 training (matched loss)": {"1B": [0.0, 0.1, -0.1], "3B": [0.1, 0.0, 0.0]},
"environment data in mid-training": {"1B": [1.1, 0.7, 1.4], "3B": [1.5, 1.2, 1.8]},
"+10% code in the mixture": {"1B": [0.9, 0.2, 1.3], "3B": [0.1, -0.4, 0.5]},
"new 200k tokenizer": {"1B": [0.5, -0.6, 0.9], "3B": [0.3, -0.2, 0.2]},
"wider attention heads": {"1B": [0.4, 0.3, 0.5], "3B": [-0.1, 0.0, 0.1]}}
NONINF = {"FP8 training (matched loss)"} # judged on "no worse" (mean > -0.2): its prize is speed
def main():
keep = []
for name, sc in RUNS.items():
cells, ok = [], True
for s in ("1B", "3B"):
m, se = st.mean(sc[s]), st.stdev(sc[s]) / len(sc[s]) ** 0.5
ok &= (m > -0.2) if name in NONINF else (m > 2 * se); cells.append(f"{s} {m:+.2f}ยฑ{se:.2f}")
verdict = ("KEEP (speed)" if name in NONINF else "KEEP") if ok else "drop"
if ok: keep.append(name)
print(f"{name:34} {' '.join(cells)} {verdict}")
print(f"\nkept {len(keep)} of {len(RUNS)}: {', '.join(keep)}")
print("FP8 passes a no-worse test, not a gain test: its prize is speed (see fp8_budget.py)")
if __name__ == "__main__":
main()Gain on the held-out suite at 1B โ 3B active parameters
Real output, bar = 3B gain. Three of the four kept ideas grow with scale; the three dropped ones shrink or are noise.
The +10% code result was the one that hurt. At 1B it was the second-best idea on the board, and two engineers had spent a month building the mixture. At 3B it was nothing.
This is why we test twice, Mira said in the review. Small models are short of everything. Give them more of anything useful and they improve. Bigger models already have enough code. The test is not whether an idea helps. It is whether it still helps when the model no longer needs help.
๐ Go deeper: what each kept idea does
- Multi-token prediction: an extra output head learns to predict two tokens ahead.
- It gives the model a richer training signal per token and can also be used for faster decoding.
- Aux-loss-free load balancing: a per-expert bias steers tokens away from overloaded experts.
- Unlike the usual extra loss term, it does not pull the model away from its main objective.
- Environment data: traces of agents solving the Part 8 environments, mixed in late in pre-training.
- The company's own advantage: no rival has these environments.
4. The ingredient that only saves money
"""fp8_budget.py โ what FP8 training buys Maestro-2.
H100 peak dense throughput doubles from BF16 (989 TFLOP/s) to FP8 (1,979).
Real end-to-end gains are far smaller: not every operation runs in FP8, and
communication and memory traffic do not speed up. The achieved speed-ups are
SCENARIO assumptions bracketing what public reports describe.
"""
FLOPS, PEAK, MFU, PRICE, GPUS = 1.06e25, 989e12, 0.40, 2.40, 4096
def main():
base = FLOPS / (PEAK * MFU * 0.91) / 3600
print(f"{'precision':26} {'GPU-hours':>10} {'days on 4,096':>14} {'cost $M':>8}")
for name, sp in (("BF16 (baseline)", 1.0), ("FP8, cautious (x1.25)", 1.25), ("FP8, planned (x1.35)", 1.35), ("FP8, ideal peak (x2.0)", 2.0)):
gh = base / sp
print(f"{name:26} {gh / 1e6:9.2f}M {gh / GPUS / 24:14.0f} {gh * PRICE / 1e6:8.1f}")
print(f"\nplanned saving: {(base - base / 1.35) * PRICE / 1e6:.1f} $M and {(base - base / 1.35) / GPUS / 24:.0f} days")
if __name__ == "__main__":
main()Days on 4,096 GPUs, and cost
Real output. The planned speed-up is 1.35ร, far below the 2ร on the data sheet, because communication and memory traffic do not get faster.
Arjun had been sceptical. He had seen FP8 runs diverge. So the team did one more thing before committing: a 3B run, matched step for step against BF16, for twice as long as the ablations. The two loss curves sat on top of each other to the third decimal place.
Five million dollars, said the finance lead, for a change nobody will ever see. That is the best kind, said Gopala.
5. Sandbox: pick the recipe
๐ฎ Sandbox: pick the recipe
Gains are the 3B-scale means from ablation_ladder.py; days and cost from fp8_budget.py. Adding gains is a simplification: real ingredients interact, which is why the kept set is re-tested together before the run.
- Try this.
- Add +10% code back in: almost no gain, one more unproven change in a $15M run.
- Untick FP8: the same model, 21 more days and $5.1M more.
- Keep only environment data: half the gain from one ingredient that nobody else has.
6. Five mistakes
- Testing ideas at one small scale and assuming they transfer.
- Using one seed and calling noise a result.
- Adding every idea that passed, without re-testing them together.
- Judging a speed change like FP8 by quality gains instead of 'no worse'.
- Keeping a result because a team spent a month on it.
7. Delegation box
- The final recipe and the kill list
- Running 42 small training runs
- Comparing loss curves
- Spare capacity on the 4,096-GPU block before the main run
8. Budget and risk tracker
| Item | Status after Part 13 | Note |
|---|---|---|
| Recipe | 4 of 7 kept | MTP, balancing, environment data, FP8 |
| Expected gain | +2.90 points at 3B | sum of kept gains, re-tested together |
| Compute | $14.5M planned | FP8 saves $5.1M and 22 days |
| FP8 risk | Watch | a matched 3B run showed no loss gap |
9. The series
The Maestro, Season 2: 23 parts in five phases (Season 1: 18 parts)
10. Reading the numbers
- Primary / standard
- DeepSeek-V3 technical report (multi-token prediction, auxiliary-loss-free balancing, FP8 training at scale); Gloeckle et al. 2024 on multi-token prediction; Micikevicius et al. 2022 on FP8 formats; NVIDIA H100 peak figures.
- Secondary โ ๏ธ
- None in this part.
- Our assumptions
- Measured gains, seeds, achieved speed-ups and all story events are the scenario.