The Pre-Training Run
Month 9. The data is frozen (Part 8), the model is designed (Part 9), the cluster heals itself (Part 7). Now 1,024 GPUs run one job for about a month, and Gopala's job changes from building to watching: is the loss going down as predicted, is the run on schedule, is that jump in the loss a blip or the start of a divergence? This part covers the learning-rate schedule, the daily rhythm of a long run, how to catch and handle loss spikes with a small detector, the run ledger that tracks tokens against plan, and the end of the run: annealing on the best data and choosing the checkpoint that goes on to post-training.
lr_schedule.py, spike_watch.py and run_ledger.py were run (real output below); spike_watch.py runs on a simulated loss curve and run_ledger.py on scenario daily numbers. run_main.sh was not executed for this article. Everything that happens in the run is the scenario. Disclosure: Claude, made by Anthropic, helps write this site.
1. Mission log: weeks 38โ46
- Checks the list in section 3. Every item green; launches on a Monday morning so the first 48 hours are watched by two people awake.
- Learning-rate decay on the annealing blend; picks the final checkpoint; hands it to post-training (Part 11).
2. What is at stake
The main run in numbers
Plan figures from Parts 3 and 7 (our assumptions).
- It is the single most expensive, least repeatable step in the plan.
- A second attempt would cost another month of the main block and most of the remaining training budget.
- So the run is designed to be resumable (checkpoints, restart wrapper) and extendable (the WSD schedule below).
- What the frontier record says.
- Meta reported 466 interruptions in 54 days for Llama 3 405B and still above 90% effective training time.
- DeepSeek reported no irrecoverable loss spikes and no rollbacks during DeepSeek-V3 pre-training.
- Both are large, experienced teams; Gopala plans for something in between.
3. The go/no-go checklist (gate G3)
| Item | Evidence | Owner |
|---|---|---|
| Data blend frozen and manifested | Part 8 manifest, hash of every shard | Gopala |
| Architecture and launch config frozen | Config in git, reviewed | Gopala |
| Throughput measured on the full block | 2-hour run at the target MFU (Part 9) | Gopala |
| Restart path tested | Fire drill (Part 7), ~11 min recovery | Gopala + contractor |
| Checkpoints save and restore | Save, kill, restore, compare loss | Gopala |
| Evaluation harness ready | Small evals run on intermediate checkpoints (Part 12) | Evaluator hat |
| On-call rota for 31 days | Two people, runbook (Part 7) | Gopala |
| Facility change freeze | Agreed with the provider (Part 5) | Gopala |
- The checklist is the gate, not a formality.
- Investors released the main-run budget at G3 (Part 3) only on these items.
- A red item delays the run; it does not get waived.
4. The learning-rate schedule
- Two common shapes.
- Cosine: warm up, then decay smoothly to a floor over the whole run. Its length must be fixed in advance.
- Warmup-stable-decay (WSD): warm up, hold the rate flat, decay only at the end (MiniCPM, 2024).
- Why he chooses WSD.
- The run can be extended if the loss is still falling well, without restarting the schedule.
- At any point he can branch a short decay off a stable checkpoint to see what a finished model would score, without stopping the main run.
- The decay phase is a natural place to switch to the annealing blend of best data.
"""lr_schedule.py โ cosine vs warmup-stable-decay (WSD) learning rates.
Cosine decays from the start, so the run length must be fixed in
advance. WSD (used by MiniCPM, 2024) holds the rate flat and decays only
at the end, so a run can be extended, and a decayed checkpoint can be
branched off at any point to check progress. Numbers are our plan.
"""
import math
PEAK, MIN_FRAC, WARM, TOTAL = 3e-4, 0.1, 2_000, 600_000 # steps (16.8M-token batches ~ 10T tokens)
DECAY_FRAC = 0.15
def cosine(s):
if s < WARM: return PEAK * s / WARM
p = (s - WARM) / (TOTAL - WARM)
return PEAK * (MIN_FRAC + (1 - MIN_FRAC) * 0.5 * (1 + math.cos(math.pi * p)))
def wsd(s):
if s < WARM: return PEAK * s / WARM
start = TOTAL * (1 - DECAY_FRAC)
if s < start: return PEAK
p = (s - start) / (TOTAL - start)
return PEAK * (MIN_FRAC + (1 - MIN_FRAC) * (1 - p))
def main():
print(f"tokens per step = {16.8e6:,.0f} -> {TOTAL*16.8e6/1e12:.1f}T tokens")
print(f"{'step':>8s} {'% run':>6s} {'cosine':>9s} {'WSD':>9s}")
for s in (0, 1_000, 2_000, 60_000, 150_000, 300_000, 450_000, 510_000, 555_000, 600_000):
print(f"{s:8,d} {s/TOTAL:6.0%} {cosine(s):9.2e} {wsd(s):9.2e}")
area = lambda f: sum(f(s) for s in range(0, TOTAL, 1000))
print(f"average LR: cosine {area(cosine)/(TOTAL/1000):.2e}, WSD {area(wsd)/(TOTAL/1000):.2e}")
if __name__ == "__main__":
main()Cosine vs WSD over the run (our plan)
x: share of the run; y: learning rate. Same formulas as lr_schedule.py.
- Reading it.
- WSD keeps a higher average learning rate (2.79e-4 vs 1.65e-4 for cosine), so it makes faster progress in the middle and does most of its "settling" in the final 15%.
5. The daily rhythm of a long run
One day of the main run
Most days nothing needs doing; the rhythm exists for the days something does.
- Dashboards answer four questions.
- Is the loss following the curve predicted from the ablation ladder (Part 9)?
- Is every rank's step time normal (stragglers, Part 7)?
- Are we on schedule (the ledger below)?
- Are evaluation scores on intermediate checkpoints rising?
- Write things down.
- A run log in git: every incident, every decision, every change. It feeds the model card (Part 14) and the next run.
6. Loss spikes: blip or divergence?
- Large runs sometimes show sudden loss jumps.
- Causes include bad batches, numerical overflow, and optimizer instability at high learning rates.
- OLMo 2's report describes changes made specifically for stability, such as a z-loss term and removing weight decay on embeddings; DeepSeek-V3 reported none that could not be recovered.
- The decision he must make quickly.
- A short spike that recovers by itself is logged and ignored.
- A spike that persists means rewinding to the last good checkpoint and skipping the batches that caused it; if it repeats, lowering the learning rate.
"""spike_watch.py โ catch loss spikes early and decide what to do.
A SIMULATED loss curve (smooth power-law decline plus noise, with two
injected spikes) is streamed through a detector that compares each step
with an exponential moving average and its running spread. Policy:
a short spike that recovers is logged; a spike that persists triggers a
rewind to the last good checkpoint and skipping the offending batches.
"""
import math, random
def loss_stream(n, seed=7):
rng = random.Random(seed)
for s in range(1, n + 1):
base = 1.9 + 9.0 * s ** -0.35
x = base + rng.gauss(0, 0.012)
if 4200 <= s < 4206: x += 0.9 # brief spike (bad batch)
if 7400 <= s < 7700: x += 0.6 * min(1, (s - 7400) / 10) # persistent divergence
yield s, x
def main():
ema, var, alpha = None, 0.0, 0.02
above, events = 0, []
for s, x in loss_stream(9000):
if ema is None: ema = x; continue
z = (x - ema) / math.sqrt(var + 1e-9) if s > 200 else 0
if z > 6:
above += 1
if above == 1: start = s
if above == 50:
events.append((start, "PERSISTENT", f"rewind to checkpoint before step {start}, skip batches {start}-{s}"))
else:
if 0 < above < 50:
events.append((start, "transient", f"{above} steps above threshold, recovered: log and continue"))
above = 0
if z < 3: # only learn from normal steps, so a spike cannot drag the baseline up
d = x - ema; ema += alpha * d; var = (1 - alpha) * (var + alpha * d * d)
for s, kind, action in events:
print(f"step {s:5d} {kind:10s} {action}")
print(f"{len(events)} events in 9,000 simulated steps")
if __name__ == "__main__":
main()- What the detector does.
- It tracks an exponential moving average of the loss and its typical noise.
- It flags a step more than six standard deviations above normal, and does not let flagged steps drag the baseline up.
- It separates the brief spike at step 4,200 (six steps) from the persistent one at step 7,401, which it escalates after 50 steps.
- A lesson from building it.
- The first version learned from the spike itself and failed to escalate the persistent one; the fix (update the baseline only on normal steps) is the kind of bug a real run would have paid for.
7. The run, as it happened
- Day 11: the loss climbs and stays up after a shard of the code mixture with corrupted tokenisation reaches the model. The detector escalates; Gopala rewinds 1,000 steps, removes the shard from the blend (recorded in the manifest) and resumes. Cost: about six hours.
- Days 1โ31: 11 hardware interruptions, all handled by the restart wrapper with hot spares; no human paged at night except on day 11.
- Day 23: a straggler alert finds one node 7% slower than its day-zero baseline (Part 6); swapped for a spare.
- These events are the story, built from the kinds of incidents published reports describe, not measurements.
Incidents during the main run (scenario)
Story values, not measurements. Part 7 expected about 15 interruptions.
8. The run ledger: on schedule?
- Tokens are the unit of progress.
- Plan: 10T tokens in 31.4 days, about 0.318T a day.
- Each morning: tokens done, ahead or behind, estimated finish, and days that fell below 90% of plan (worth investigating).
"""run_ledger.py โ is the main run on schedule?
Daily check of tokens trained against plan: the plan (Part 3) is 10T
tokens in ~31.4 days on 1,024 GPUs. The DAILY numbers below are
SCENARIO values written for this story, not measurements.
"""
PLAN_TOKENS, PLAN_DAYS = 10e12, 31.4
DAILY = [0.31, 0.33, 0.33, 0.30, 0.33, 0.33, 0.24, 0.33, 0.33, 0.32, # trillions of tokens per day
0.33, 0.33, 0.33, 0.29, 0.33, 0.33, 0.33, 0.33, 0.26, 0.33]
def main():
rate = PLAN_TOKENS / PLAN_DAYS / 1e12
done = 0
print(f"plan: {rate:.3f}T tokens/day")
for d, t in enumerate(DAILY, 1):
done += t
if d % 5 == 0:
ahead = done - rate * d
eta = d + (PLAN_TOKENS / 1e12 - done) / (done / d)
print(f"day {d:2d}: {done:5.2f}T done ({done/(PLAN_TOKENS/1e12):4.0%}), "
f"{'ahead' if ahead >= 0 else 'behind'} by {abs(ahead):.2f}T, finish ~day {eta:.1f}")
low = [d for d, t in enumerate(DAILY, 1) if t < 0.9 * rate]
print("days below 90% of plan (investigate):", ", ".join(map(str, low)))
if __name__ == "__main__":
main()- Reading it.
- The run stays within 0.03T tokens of plan; the projected finish moves between day 31.2 and 31.7.
- Days 7 and 19 were slow: a planned facility test that overlapped a checkpoint (day 7) and the day-11-style rewind repeated on a smaller scale (day 19 in the story).
9. The end: annealing and choosing a checkpoint
- The last 15% of the run is different.
- The learning rate decays (WSD).
- The blend switches towards the highest-quality data: the top quality-classifier scores, maths, code, and the capped verified-skill data from Part 8.
- Llama 3's report describes a similar annealing phase on high-quality data at the end of pre-training.
- Choosing the checkpoint.
- Not simply the last one: he compares the last few on the held-out evaluation suite and on contamination-checked benchmarks (Part 12).
- The chosen checkpoint, its config, data manifest and run log are frozen together as "Maestro-1 base".
- A small decayed branch earlier in the run tells him what the final model will look like.
- Thanks to WSD, he branched a short decay at 60% of the run to sanity-check scores before committing the last weeks.
From the main run to the base model
The base model is the input to Part 11.
10. The restart wrapper
- The automation that makes a month-long run survivable for one person.
- It relaunches from the last good checkpoint, excludes drained nodes, and stops after three restarts in an hour so a human looks first (Part 7's policy).
#!/bin/bash
# run_main.sh โ the restart wrapper around the main run (Part 7's policy).
# Requeues from the latest good checkpoint, excluding drained nodes,
# and stops after 3 restarts in an hour so a human looks first.
set -u
RESTARTS=()
while true; do
BAD=$(sinfo -h -N -t drain,down -o %N | paste -sd, -)
sbatch --wait --exclude="${BAD}" launch_megatron.sh --load /checkpoints/maestro-1
rc=$?
[ $rc -eq 0 ] && { echo "run finished"; exit 0; }
now=$(date +%s); RESTARTS+=($now)
recent=$(for t in "${RESTARTS[@]}"; do [ $((now - t)) -lt 3600 ] && echo x; done | wc -l)
[ "$recent" -ge 3 ] && { echo "3 restarts in 1 h: paging a human"; ./page.sh "main run unstable"; exit 1; }
sleep 60
done11. What the run cost
- In the story the run finishes in 32.5 days (one day over plan because of the day-11 rewind and slow days) at about 32% MFU, using about 1.86M dollars of the reserved block's time against the plan's $1.81M.
- Because the block is reserved for 12 months (Part 4), the real cost of the overrun is a day of the block not available for other work, not extra cash.
- These are story values.
12. Evaluating while it trains
- Do not wait 31 days to learn the model is bad.
- Every few hundred billion tokens, a small fixed evaluation suite runs on the latest checkpoint on spare GPUs: language modelling loss on held-out data, a few multiple-choice knowledge tests, short code tasks with unit tests.
- Early scores are noisy and low.
- Base models before annealing score far below their final level on reasoning tasks; the trend matters more than the value.
- He compares against the curve predicted from the ablation ladder (Part 9), not against finished models.
- The decayed branch at 60%.
- Because WSD makes it cheap, he decays a branch for a few thousand steps at 60% of the run and evaluates it like a finished model; that is the best early warning of the final quality.
What is checked, and how often
Cadence from Gopala's run plan (our design).
13. After the run: extending the context
- Pre-training ran at 8,192 tokens; clients need longer.
- Engineering tasks involve long files and logs.
- A short extra phase extends it.
- Continue training on long documents with an adjusted rotary position base, increasing the sequence length in steps.
- Llama 3's report describes extending context in stages at the end of pre-training, up to 128K tokens.
- He targets 64K first.
- It is cheaper, covers most repository-level tasks, and can be extended later.
- Checked with retrieval tests ("find the needle") and, more importantly, real long-file coding tasks.
14. What gets frozen with the base model
- A model without its records cannot be trusted later.
- The weights of the chosen checkpoint, plus the two before it.
- The exact launch configuration and code version.
- The data manifest, including the shard removed on day 11.
- The run log: incidents, decisions, ledger, evaluation history.
- The tokenizer and its training sample.
- Where it goes.
- Two storage locations with different providers, write-once, with hashes checked monthly.
- Part 14's approvals and the model card are written from these records.
15. Seven run mistakes
- Launching on a Friday.
- Choosing a schedule whose length cannot change.
- Letting a spike detector learn from the spike.
- Changing the data blend without recording it.
- Waiting for the end of the run to evaluate anything.
- Picking the last checkpoint by default.
- One person on call for a month.
16. Delegation box
- Go/no-go, rewind and checkpoint decisions
- Morning reviews
- Choosing the final checkpoint
- Overnight incident summaries (read-only)
- Daily ledger and eval reports
- Searching logs for the cause of slow days
- The provider's on-site team
- A contractor sharing night on-call
- Open-source training framework and tools
17. Budget and risk tracker
| Item | Status after Part 10 | Note |
|---|---|---|
| Spend to date | ~$5.6M (scenario) | Main run, ablations, serving Maestro-0 |
| Model | Maestro-1 base frozen (scenario) | 10T tokens, 32.5 days |
| Time | Week 46 (month ~10.5) | On plan within a few days |
| Training risk | Retired | Run completed |
| Quality risk | Open | Base model scores to be judged after post-training and evals |
| Budget risk | Watch | About $2.5M of training compute left for post-training and Maestro-1.1 |
18. The series
The Maestro: 18 parts in five phases
- The main run is boring by design: a schedule that can stretch, a detector that knows a blip from a divergence, a ledger that says where the run stands every morning, and a restart path that handles the hardware without waking anyone.
- One real incident cost six hours instead of a week.
- Next, Part 11: turning the base model into Maestro-1, with the self-improving loop and its gate.
19. Reading the numbers
- Primary / standard
- Llama 3: 466 interruptions in 54 days, >90% effective training time, annealing on high-quality data at the end of pre-training (Meta). DeepSeek-V3: no irrecoverable loss spikes or rollbacks (DeepSeek). OLMo 2 stability changes, including z-loss (AI2). WSD schedule (MiniCPM, Hu et al., 2024).
- Secondary โ ๏ธ
- None in this part.
- Our assumptions
- Learning rates, batch size, the simulated loss curve, daily token numbers, all incidents and the run outcome.