Shipping an Experiment From Idea to Merged PR

The Whole Job, in One Pass

Seven articles in this series each answered a narrow question well: how do you read production training code, what happens when a distributed job won't converge, how does compute get allocated, when do you write a custom kernel, what does on-call feel like, how do you work with other teams' code. In reality, a single meaningfully-sized experiment touches almost all of these in sequence, usually within the span of one to three weeks. This article is deliberately structured as a single realistic timeline rather than another set of independent parts, because the compounding — not any one piece — is the actual skill this whole series has been building toward.

A Hypothesis, Not Just a Notion

The experiment: a research engineer suspects a specific change to the data-mixing ratio between two training corpora will improve downstream evaluation performance without hurting training stability. Before any code changes, the idea is written up as a falsifiable hypothesis with a specific, cheap first test — following Article 3 of this series' instinct to isolate before scaling: run the change on a small model and short training run first, and only commit larger compute (Article 4) once the small-scale signal looks real.

A New Config, Not a New Training Loop

Following Article 2 of this series directly, the change is implemented as a new configuration override on top of the shared base config — a new data-mixing ratio parameter — rather than a fork of the training loop itself. This is what makes the experiment mergeable at all: a config-only change is trivially reviewable and diffable, while a training-loop fork would immediately trigger the exact code-divergence problem Sculley et al.'s paper warns about, and would need to be reconciled with the shared loop before it could ever be merged.

Right-Sizing the Ask

Per Article 4, the small-scale validation run is requested on a modest quota, quick-turnaround queue rather than the team's premium reserved capacity — the hypothesis hasn't earned that budget yet. Only after the small run shows a real, reproducible signal does the engineer request a larger allocation, and even then on spot/preemptible capacity with proactive checkpointing (Article 4, Part 4's etiquette), since a mid-scale validation run can tolerate an occasional restart in a way a final release candidate could not.

When the First Launch Doesn't Just Work

The larger run fails to start on the first attempt — a environment mismatch between the small-scale and larger-scale launch configs. Article 3's "won't start" triage category applies directly: the fix is in environment and scheduling, not in the model or the data-mixing logic itself, and confirming that distinction in the first few minutes avoids a wasted afternoon debugging the wrong layer. Once running, loss curves are sane but a second issue surfaces: aggregate GPU utilization looks fine, but Article 4's Model FLOPs Utilization metric is unexpectedly low — tracing it to a data-loading bottleneck introduced by the new mixing logic, not a compute problem at all.

Watching Without Staring

With the fix deployed, the full-scale run is launched with alerting configured per Article 6's discipline — a page only for genuinely actionable conditions (loss divergence, node failure), a dashboard for everything else, avoiding the alert-fatigue trap while still catching a real failure quickly if one occurs overnight. The engineer is not paged during this run; the multi-day training completes cleanly, which is itself worth noting as the unremarkable, desired outcome of Article 6's entire framework.

Turning Results Into a Mergeable Change

With results in hand, the actual code change (still just the config diff from Part 2, plus a small data-loading fix from Part 4) is opened as a pull request. Article 1's blast-radius principle applies directly to review depth: the config diff itself is narrow and low-risk, but the data-loading fix touches a shared utility another team also depends on — triggering exactly the cross-team etiquette from Article 7, where the owning team is looped in before the fix merges, not after, and the PR description explains the motivating experiment so the reviewer has real context rather than just a diff.

Every Article, in One Experiment

Readiness Checklist

⚠️ What's Missing or Uncertain

Where This Series Goes Next

Article 9, the capstone of this series, steps back from any single experiment to describe a full week in the life of a research engineer — showing how these pieces interleave with meetings, other people's incidents, and the ordinary unpredictability of a real job, not just one clean linear story.

🎥 Recommended Videos

🧭 Closing — The Job Is the Sequence, Not Any One Step