Home›Blog›Frontier Lab Engineering — Production Training Code
Frontier Lab Engineering Practicum · Article 2 of 9📜
Reading and Writing Production Training Code
nanoGPT fits in one file. A real team's training code doesn't, and can't — dozens of engineers need to run hundreds of variants of "roughly the same" training job without duplicating a training loop hundreds of times. This article covers how configs, launchers, and naming conventions actually solve that problem, and how to read someone else's setup without getting lost.
The Foundations series' reproduction exercises (nanoGPT, a single-file training script) are the right way to learn the mechanics of a training loop. They are also nothing like what a team of engineers runs day to day, and the gap between the two is itself worth understanding before your first week. A single script, hard-coded hyperparameters and all, works fine for one person running one experiment. It breaks down the moment ten engineers need to run two hundred variants of "the same" experiment — different learning rates, different data mixes, different model sizes — without two hundred slightly-diverged copies of the training loop, each with its own silent bugs.
This article covers the three concrete mechanisms production ML teams use to solve that problem: configuration composition, launchers, and naming conventions — plus where, deliberately, the underlying training code must stay identical no matter how many configs sit on top of it.
200+Realistic number of experiment variants a team might run off one base training script in a month
1Training loop that should exist per project — everything else should be configuration, not duplication
2015Year Sculley et al.'s "Hidden Technical Debt in Machine Learning Systems" named config sprawl as a real, citable engineering problem
0Lines of the actual training loop a well-designed config change should ever require touching
Part 1 — Configuration Composition
One Base Config, Many Small Overrides
The core idea, used across most production ML codebases: define one base configuration file with sensible defaults for every setting a training run needs (learning rate, batch size, model size, dataset path), then let individual experiments override only the handful of values they actually want to change, rather than copying the entire configuration. Hydra (Meta's open-source configuration framework, built on OmegaConf) is the most widely adopted tool implementing exactly this pattern in ML research code.
Each experiment overrides only the one or two values it actually wants to change — everything else silently inherits from the base config, so there's no copy-paste drift between runs.
python train.py --config=basemodel_size=7Bdataset=v2_mix# everything else (lr, batch_size, optimizer...) comes from base.yaml unchanged
This is precisely the config discipline that made Chinchilla-style ablations (Article 1 and Article 3 of the Foundations series) practical to run at scale: a real compute-optimal sweep means running the same training loop dozens of times with only the model-size and data-quantity settings changing — untenable if each run required its own hand-edited copy of the script.
Part 2 — Launchers
Decoupling "What to Run" From "Where It Runs"
A second, related separation: the code that defines what a training run does (the model, the loss, the optimizer step) should be entirely independent of the code that decides where it runs — on your laptop for a two-minute debug run, or across a thousand-GPU cluster via the job scheduler covered in Article 10 of the Foundations series. A "launcher" script handles this translation, taking the same config and training entry point and wrapping it with whatever cluster-specific job submission logic (Slurm flags, container images, resource requests) the environment needs.
The same training code, unmodified, runs under two very different launchers — the model and loop never need to know or care which one is in use.
Part 3 — Experiment Naming and Tracking Conventions
Why Naming a Run Well Is a Real Engineering Skill
Article 2 of the Foundations series introduced Weights & Biases as an individual habit ("log every run"). At team scale, with hundreds of runs across many people, a consistent naming and tagging convention is what makes that logging actually useful six weeks later, rather than an unsearchable pile.
1
Encode the variables that changed, not just a date. A run named run_sept15 tells you nothing later; a run named 7b_v2data_lr1e4 tells you exactly what this run is testing, without opening the config.
2
Tag by purpose, not just content. Separating "exploratory" runs from "candidate for the paper/release" runs prevents a stray debug run from being mistaken for a real result weeks later.
3
Link the run back to the exact config and code commit. A run is only reproducible (Article 2's core lesson) if you can trace it back to the precise config override and git commit that produced it — most experiment trackers support logging this automatically, but only if you set it up before, not after, you need it.
Part 4 — Where the Code Must Not Diverge
Configuration Flexibility Has a Boundary
Parts 1–3 all assume one thing: the actual training loop — forward pass, loss computation, backward pass, optimizer step (Article 2 of the Foundations series' first self-assessment item) — stays a single, shared piece of code that every config and launcher points at, never duplicated or hand-modified per experiment. The moment someone copies the training loop itself to try a quick variant, the codebase now has two loops that will silently drift apart, and a bug fixed in one will not exist in the other. This is precisely the "config debt" and "glue code" problem Sculley et al.'s "Hidden Technical Debt in Machine Learning Systems" (2015) named directly — one of the most cited papers in applied ML engineering precisely because this failure mode is so common and so expensive to unwind later.
A good config system doesn't just make experiments easier to run — it makes duplicating the training loop unnecessary, which is the actual mechanism protecting the codebase from the technical debt Sculley et al. warned about a decade ago.
Part 5 — Real Scenarios
Real Scenario Walkthroughs
📖Scenario A — Reading an Unfamiliar Config for the First Time
A realistic first-week task from Article 1 of this series: understand what a colleague's experiment config actually does before touching it. The efficient approach, given Part 1's composition pattern, is never to read the override file in isolation — trace it back to the base config it inherits from, then read only the diff between the two. Trying to understand an override file as if it were complete, self-contained code is a common, avoidable source of new-hire confusion.
The lesson: in a composed config system, the override file is intentionally incomplete — understanding it requires reading the inheritance chain, not just the file in front of you.
🚫Scenario B — Resisting the Urge to Copy the Training Script
A very common early mistake: needing to test one new idea (a different loss term, say) and, rather than learning how to properly extend the shared training loop, copying the whole script into a new file and hacking in the change. This works in the short term and creates exactly the two-diverging-loops problem Part 4 warns about — a bug fix or improvement to the "real" loop will silently never reach the copy, and vice versa.
The lesson: the extra 30 minutes spent learning how to add a config-driven option to the shared loop is almost always cheaper than the technical debt of a forked copy.
Readiness Checklist
1
Have you used a configuration framework (Hydra or similar) where a single small override file changes only a few values from a shared base?
2
Can you explain why a training loop should stay identical across a local debug run and a large cluster run, and what the launcher is responsible for instead?
3
Have you named and tagged an experiment run so that a colleague (or you, six weeks later) could understand what it tested without opening its config?
4
Have you ever resisted (or regretted not resisting) the urge to copy a training script rather than extending it with a config option?
⚠️ What's Missing or Uncertain
This article describes general patterns, not one canonical setup. Specific configuration tools vary between organizations — some teams build fully custom config systems rather than using Hydra — but the underlying principle (one base config, small composable overrides, a training loop that never forks) is consistent across essentially every production ML codebase this article is aware of.
Where This Series Goes Next
Article 3 moves from writing training code to debugging it when it breaks — a practical runbook for a distributed job that won't converge or won't even start, building directly on Article 10 of the Foundations series' parallelism theory and this article's config/launcher vocabulary to actually diagnose real failures.
🧭 Closing — Flexibility on Top, Discipline Underneath
🎯 The Bottom Line
Production training code looks different from a solo reproduction script not because the underlying training loop is more complicated, but because a whole layer — config composition, launchers, naming conventions — exists specifically to let many people run many experiments against one shared, undivided training loop. The discipline of never forking that shared loop is what makes hundreds of daily experiments manageable instead of an unmaintainable mess — precisely the technical debt Sculley et al. named a decade before most people writing ML training code today started their careers.