The On-Call Reality: Monitoring, Alerts, and 2am Incidents

The Job Doesn't End When the Job Starts Training

Every article in this series so far has focused on getting a system working: writing the code, requesting the compute, debugging the failure, optimizing the kernel. None of that guarantees the system keeps working unattended, at 2am, when nobody is watching a terminal. Google's Site Reliability Engineering book — already cited in Article 1 of this series for its monorepo and tooling context — devotes an entire section to exactly this problem: how do you build confidence that a system will either work correctly while unattended, or fail loudly enough that a human finds out in time to matter. On-call is the organizational answer to that question, and it is one of the least-discussed but most universal parts of working at any lab or company running production systems, training or otherwise.

Beyond "Is It Running"

A naive monitoring setup checks whether a process is alive. A real one, following the SRE book's framing of the "four golden signals" (latency, traffic, errors, saturation), checks whether the system is doing its job correctly and efficiently — which for a training job maps directly onto concepts this series has already built: Article 4's Model FLOPs Utilization (is the GPU doing useful work, not just running), Article 3's loss-curve sanity checks (is training actually converging, not just producing numbers), and basic infrastructure health (disk, network, node failures). The uncomfortable truth many new engineers learn the hard way is that a job can be "alive" by every naive check while being completely useless by every meaningful one — the exact "busy but starved" pattern from Article 4's Scenario A, except now happening silently, unattended, overnight.

The Alert That Cried Wolf

Rob Ewaschuk's widely-circulated internal Google document, later published as the SRE book's "Monitoring Distributed Systems" chapter, states the core alerting principle plainly: every alert that pages a human should require immediate human action, and every alert that doesn't should not page anyone. Violating this in either direction has a real cost. Too many low-value alerts trains the on-call engineer to reflexively dismiss pages — the alert fatigue problem — so that the one alert that actually matters gets the same tired, skeptical response as the hundred before it that didn't. Too few alerts, or alerts tuned only to catastrophic failure, means smaller but still-costly problems (a slow leak in GPU utilization, a data pipeline quietly dropping a fraction of batches) go undetected for days.

From Page to Resolution

A real incident, however dramatic it feels at 2am, tends to follow a repeatable shape: acknowledge the page, assess severity (is this degraded but tolerable, or actively losing data/progress), mitigate first and root-cause later (restore service, even imperfectly, before fully understanding why it broke — directly echoing Article 3's triage instinct of shrinking the problem before solving it completely), and only then investigate the underlying cause once the immediate fire is out. A runbook — a written, specific procedure for a known failure mode, distinct from general documentation — exists precisely so this sequence doesn't depend on the on-call engineer's memory or the original author being awake and reachable; PagerDuty's widely-used Incident Response documentation formalizes this same shape as an industry-standard practice, not something unique to any one company.

Why "Blameless" Isn't Just a Nice Word

John Allspaw's influential Etsy engineering post, "Blameless PostMortems and a Just Culture" (2012), makes the case that assigning individual blame after an incident actively makes future incidents more likely, not less — because engineers who fear blame learn to hide near-misses and avoid touching risky systems, rather than surfacing the information that would actually prevent a repeat. A blameless postmortem instead treats the incident as a symptom of the system (the alert that didn't fire soon enough, the runbook that didn't cover this case, the monitoring gap that let it go undetected) rather than of the individual who happened to be on call or who wrote the change that triggered it. This directly extends Article 2 of this series' point about production training code: a bug that reaches production is usually evidence of a missing safeguard in the system, not solely a mistake by whoever wrote the line.

The Human Cost Nobody Puts on a Roadmap

The SRE book is explicit that on-call load itself is a system health metric worth tracking — a rotation that pages someone every night is not a sustainable steady state, it's a signal that either monitoring is miscalibrated (Part 2) or the underlying system has an unresolved reliability problem. Reasonable rotation size, defined escalation paths so no single person is the only one who can respond, and treating a high page volume as an engineering priority rather than an accepted cost of doing business are what separate teams that retain engineers from teams that burn them out.

Real Scenario Walkthroughs

Readiness Checklist

⚠️ What's Missing or Uncertain

Where This Series Goes Next

Article 7 moves from responding to your own team's incidents to a different daily friction: working with — and around — other teams' code, covering shared library ownership, cross-team dependencies, and the etiquette of changing code you don't fully understand the history of.

🎥 Recommended Videos

🧭 Closing — Reliability Is a Practice, Not an Accident