Home โ€บ Blog โ€บ The Maestro ยท Part 10
The Maestro ยท Part 10 ยท Phase 2 ยท ๐Ÿง  ML engineer hat

The Pre-Training Run

Month 9. The data is frozen (Part 8), the model is designed (Part 9), the cluster heals itself (Part 7). Now 1,024 GPUs run one job for about a month, and Gopala's job changes from building to watching: is the loss going down as predicted, is the run on schedule, is that jump in the loss a blip or the start of a divergence? This part covers the learning-rate schedule, the daily rhythm of a long run, how to catch and handle loss spikes with a small detector, the run ledger that tracks tokens against plan, and the end of the run: annealing on the best data and choosing the checkpoint that goes on to post-training.

FL
FrontierAGI Team
๐Ÿง  ML engineer๐Ÿ› ๏ธ Operator๐Ÿ“ˆ Analyst๐Ÿ’ป Coder
โš ๏ธ How to read this (as of 30 Oct 2026) Llama 3's interruption figures, OLMo 2's stability techniques, DeepSeek-V3's stability report and the WSD schedule (MiniCPM) are from their papers. lr_schedule.py, spike_watch.py and run_ledger.py were run (real output below); spike_watch.py runs on a simulated loss curve and run_ledger.py on scenario daily numbers. run_main.sh was not executed for this article. Everything that happens in the run is the scenario. Disclosure: Claude, made by Anthropic, helps write this site.

1. Mission log: weeks 38โ€“46

Week 38 ยท Go/no-go (gate G3)
  • Checks the list in section 3. Every item green; launches on a Monday morning so the first 48 hours are watched by two people awake.
Weeks 39โ€“42 ยท The long middle
  • Daily rhythm (section 5). One persistent loss spike handled by rewinding (section 7); 11 hardware interruptions, all recovered automatically (Part 7).
Weeks 43โ€“46 ยท Decay and finish
  • Learning-rate decay on the annealing blend; picks the final checkpoint; hands it to post-training (Part 11).

2. What is at stake

The main run in numbers

10Ttokens planned
~31 dayson 1,024 GPUs (Part 3)
$1.81Mfinal-run compute (Part 3, our model)
~15interruptions expected (Part 7)

Plan figures from Parts 3 and 7 (our assumptions).

3. The go/no-go checklist (gate G3)

ItemEvidenceOwner
Data blend frozen and manifestedPart 8 manifest, hash of every shardGopala
Architecture and launch config frozenConfig in git, reviewedGopala
Throughput measured on the full block2-hour run at the target MFU (Part 9)Gopala
Restart path testedFire drill (Part 7), ~11 min recoveryGopala + contractor
Checkpoints save and restoreSave, kill, restore, compare lossGopala
Evaluation harness readySmall evals run on intermediate checkpoints (Part 12)Evaluator hat
On-call rota for 31 daysTwo people, runbook (Part 7)Gopala
Facility change freezeAgreed with the provider (Part 5)Gopala

4. The learning-rate schedule

lr_schedule.py ยท Python 3, no dependencies
"""lr_schedule.py โ€” cosine vs warmup-stable-decay (WSD) learning rates.

Cosine decays from the start, so the run length must be fixed in
advance. WSD (used by MiniCPM, 2024) holds the rate flat and decays only
at the end, so a run can be extended, and a decayed checkpoint can be
branched off at any point to check progress. Numbers are our plan.
"""
import math

PEAK, MIN_FRAC, WARM, TOTAL = 3e-4, 0.1, 2_000, 600_000   # steps (16.8M-token batches ~ 10T tokens)
DECAY_FRAC = 0.15

def cosine(s):
    if s < WARM: return PEAK * s / WARM
    p = (s - WARM) / (TOTAL - WARM)
    return PEAK * (MIN_FRAC + (1 - MIN_FRAC) * 0.5 * (1 + math.cos(math.pi * p)))

def wsd(s):
    if s < WARM: return PEAK * s / WARM
    start = TOTAL * (1 - DECAY_FRAC)
    if s < start: return PEAK
    p = (s - start) / (TOTAL - start)
    return PEAK * (MIN_FRAC + (1 - MIN_FRAC) * (1 - p))

def main():
    print(f"tokens per step = {16.8e6:,.0f} -> {TOTAL*16.8e6/1e12:.1f}T tokens")
    print(f"{'step':>8s} {'% run':>6s} {'cosine':>9s} {'WSD':>9s}")
    for s in (0, 1_000, 2_000, 60_000, 150_000, 300_000, 450_000, 510_000, 555_000, 600_000):
        print(f"{s:8,d} {s/TOTAL:6.0%} {cosine(s):9.2e} {wsd(s):9.2e}")
    area = lambda f: sum(f(s) for s in range(0, TOTAL, 1000))
    print(f"average LR: cosine {area(cosine)/(TOTAL/1000):.2e}, WSD {area(wsd)/(TOTAL/1000):.2e}")

if __name__ == "__main__":
    main()
$ python3 lr_schedule.py tokens per step = 16,800,000 -> 10.1T tokens step % run cosine WSD 0 0% 0.00e+00 0.00e+00 1,000 0% 1.50e-04 1.50e-04 2,000 0% 3.00e-04 3.00e-04 60,000 10% 2.94e-04 3.00e-04 150,000 25% 2.61e-04 3.00e-04 300,000 50% 1.66e-04 3.00e-04 450,000 75% 6.98e-05 3.00e-04 510,000 85% 4.48e-05 3.00e-04 555,000 92% 3.38e-05 1.65e-04 600,000 100% 3.00e-05 3.00e-05 average LR: cosine 1.65e-04, WSD 2.79e-04

Cosine vs WSD over the run (our plan)

x: share of the run; y: learning rate. Same formulas as lr_schedule.py.

5. The daily rhythm of a long run

One day of the main run

๐ŸŒ…Morning reviewLoss, step time, tokens vs plan, overnight incidents
๐Ÿ“Eval checkSmall benchmark suite on the latest checkpoint (Part 12)
๐Ÿ› ๏ธFix and fileTickets for drained nodes; Xid table updates
๐ŸŒ™HandoverRunbook check with the night on-call

Most days nothing needs doing; the rhythm exists for the days something does.

6. Loss spikes: blip or divergence?

spike_watch.py ยท Python 3, no dependencies; simulated loss curve
"""spike_watch.py โ€” catch loss spikes early and decide what to do.

A SIMULATED loss curve (smooth power-law decline plus noise, with two
injected spikes) is streamed through a detector that compares each step
with an exponential moving average and its running spread. Policy:
a short spike that recovers is logged; a spike that persists triggers a
rewind to the last good checkpoint and skipping the offending batches.
"""
import math, random

def loss_stream(n, seed=7):
    rng = random.Random(seed)
    for s in range(1, n + 1):
        base = 1.9 + 9.0 * s ** -0.35
        x = base + rng.gauss(0, 0.012)
        if 4200 <= s < 4206: x += 0.9           # brief spike (bad batch)
        if 7400 <= s < 7700: x += 0.6 * min(1, (s - 7400) / 10)   # persistent divergence
        yield s, x

def main():
    ema, var, alpha = None, 0.0, 0.02
    above, events = 0, []
    for s, x in loss_stream(9000):
        if ema is None: ema = x; continue
        z = (x - ema) / math.sqrt(var + 1e-9) if s > 200 else 0
        if z > 6:
            above += 1
            if above == 1: start = s
            if above == 50:
                events.append((start, "PERSISTENT", f"rewind to checkpoint before step {start}, skip batches {start}-{s}"))
        else:
            if 0 < above < 50:
                events.append((start, "transient", f"{above} steps above threshold, recovered: log and continue"))
            above = 0
            if z < 3:   # only learn from normal steps, so a spike cannot drag the baseline up
                d = x - ema; ema += alpha * d; var = (1 - alpha) * (var + alpha * d * d)
    for s, kind, action in events:
        print(f"step {s:5d}  {kind:10s} {action}")
    print(f"{len(events)} events in 9,000 simulated steps")

if __name__ == "__main__":
    main()
$ python3 spike_watch.py # simulated loss curve with two injected spikes step 4200 transient 6 steps above threshold, recovered: log and continue step 7401 PERSISTENT rewind to checkpoint before step 7401, skip batches 7401-7450 2 events in 9,000 simulated steps

7. The run, as it happened

Scenario
  • Day 11: the loss climbs and stays up after a shard of the code mixture with corrupted tokenisation reaches the model. The detector escalates; Gopala rewinds 1,000 steps, removes the shard from the blend (recorded in the manifest) and resumes. Cost: about six hours.
  • Days 1โ€“31: 11 hardware interruptions, all handled by the restart wrapper with hot spares; no human paged at night except on day 11.
  • Day 23: a straggler alert finds one node 7% slower than its day-zero baseline (Part 6); swapped for a spare.
  • These events are the story, built from the kinds of incidents published reports describe, not measurements.

Incidents during the main run (scenario)

Hardware interruptions, auto-recovered11
Persistent loss spike, rewound1
Straggler node swapped1
Pages to a human at night1

Story values, not measurements. Part 7 expected about 15 interruptions.

8. The run ledger: on schedule?

run_ledger.py ยท Python 3, no dependencies; scenario daily numbers
"""run_ledger.py โ€” is the main run on schedule?

Daily check of tokens trained against plan: the plan (Part 3) is 10T
tokens in ~31.4 days on 1,024 GPUs. The DAILY numbers below are
SCENARIO values written for this story, not measurements.
"""
PLAN_TOKENS, PLAN_DAYS = 10e12, 31.4
DAILY = [0.31, 0.33, 0.33, 0.30, 0.33, 0.33, 0.24, 0.33, 0.33, 0.32,   # trillions of tokens per day
         0.33, 0.33, 0.33, 0.29, 0.33, 0.33, 0.33, 0.33, 0.26, 0.33]

def main():
    rate = PLAN_TOKENS / PLAN_DAYS / 1e12
    done = 0
    print(f"plan: {rate:.3f}T tokens/day")
    for d, t in enumerate(DAILY, 1):
        done += t
        if d % 5 == 0:
            ahead = done - rate * d
            eta = d + (PLAN_TOKENS / 1e12 - done) / (done / d)
            print(f"day {d:2d}: {done:5.2f}T done ({done/(PLAN_TOKENS/1e12):4.0%}), "
                  f"{'ahead' if ahead >= 0 else 'behind'} by {abs(ahead):.2f}T, finish ~day {eta:.1f}")
    low = [d for d, t in enumerate(DAILY, 1) if t < 0.9 * rate]
    print("days below 90% of plan (investigate):", ", ".join(map(str, low)))

if __name__ == "__main__":
    main()
$ python3 run_ledger.py # scenario numbers for the first 20 days plan: 0.318T tokens/day day 5: 1.60T done ( 16%), ahead by 0.01T, finish ~day 31.2 day 10: 3.15T done ( 32%), behind by 0.03T, finish ~day 31.7 day 15: 4.76T done ( 48%), behind by 0.02T, finish ~day 31.5 day 20: 6.34T done ( 63%), behind by 0.03T, finish ~day 31.5 days below 90% of plan (investigate): 7, 19

9. The end: annealing and choosing a checkpoint

From the main run to the base model

๐Ÿ‹๏ธStable phase85% of steps at peak LR
๐Ÿ“‰Decay phaseLR to 10% on the annealing blend
๐ŸงชCompareLast checkpoints on held-out evals
๐ŸงŠFreezeMaestro-1 base + manifest + run log

The base model is the input to Part 11.

10. The restart wrapper

run_main.sh ยท Slurm restart wrapper ยท not executed for this article
#!/bin/bash
# run_main.sh โ€” the restart wrapper around the main run (Part 7's policy).
# Requeues from the latest good checkpoint, excluding drained nodes,
# and stops after 3 restarts in an hour so a human looks first.
set -u
RESTARTS=()
while true; do
  BAD=$(sinfo -h -N -t drain,down -o %N | paste -sd, -)
  sbatch --wait --exclude="${BAD}" launch_megatron.sh --load /checkpoints/maestro-1
  rc=$?
  [ $rc -eq 0 ] && { echo "run finished"; exit 0; }
  now=$(date +%s); RESTARTS+=($now)
  recent=$(for t in "${RESTARTS[@]}"; do [ $((now - t)) -lt 3600 ] && echo x; done | wc -l)
  [ "$recent" -ge 3 ] && { echo "3 restarts in 1 h: paging a human"; ./page.sh "main run unstable"; exit 1; }
  sleep 60
done

11. What the run cost

Scenario
  • In the story the run finishes in 32.5 days (one day over plan because of the day-11 rewind and slow days) at about 32% MFU, using about 1.86M dollars of the reserved block's time against the plan's $1.81M.
  • Because the block is reserved for 12 months (Part 4), the real cost of the overrun is a day of the block not available for other work, not extra cash.
  • These are story values.

12. Evaluating while it trains

What is checked, and how often

Every stepLoss, gradient norm, step time per rank
Every hourTokens vs plan, node health, data loader throughput
Every ~300B tokensSmall evaluation suite on the latest checkpoint
Once, at ~60%Decayed branch evaluated like a finished model

Cadence from Gopala's run plan (our design).

13. After the run: extending the context

14. What gets frozen with the base model

15. Seven run mistakes

16. Delegation box

๐Ÿ™‹ Gopala himself
  • Go/no-go, rewind and checkpoint decisions
  • Morning reviews
  • Choosing the final checkpoint
๐Ÿค– AI agents
  • Overnight incident summaries (read-only)
  • Daily ledger and eval reports
  • Searching logs for the cause of slow days
๐Ÿค Bought, rented or borrowed
  • The provider's on-site team
  • A contractor sharing night on-call
  • Open-source training framework and tools

17. Budget and risk tracker

ItemStatus after Part 10Note
Spend to date~$5.6M (scenario)Main run, ablations, serving Maestro-0
ModelMaestro-1 base frozen (scenario)10T tokens, 32.5 days
TimeWeek 46 (month ~10.5)On plan within a few days
Training riskRetiredRun completed
Quality riskOpenBase model scores to be judged after post-training and evals
Budget riskWatchAbout $2.5M of training compute left for post-training and Maestro-1.1

18. The series

The Maestro: 18 parts in five phases

Verdict
  • The main run is boring by design: a schedule that can stretch, a detector that knows a blip from a divergence, a ledger that says where the run stands every morning, and a restart path that handles the hardware without waking anyone.
    • One real incident cost six hours instead of a week.
    • Next, Part 11: turning the base model into Maestro-1, with the self-improving loop and its gate.

19. Reading the numbers

Primary / standard
Llama 3: 466 interruptions in 54 days, >90% effective training time, annealing on high-quality data at the end of pre-training (Meta). DeepSeek-V3: no irrecoverable loss spikes or rollbacks (DeepSeek). OLMo 2 stability changes, including z-loss (AI2). WSD schedule (MiniCPM, Hu et al., 2024).
Secondary โš ๏ธ
None in this part.
Our assumptions
Learning rates, batch size, the simulated loss curve, daily token numbers, all incidents and the run outcome.