Home โ€บ Blog โ€บ Observability and On-Call
Who Runs the AI Factory ยท Part 11 ยท Observability & on-call

Who Watches the Factory? Observability, Incidents and the SREs Behind AI Services

Every layer in this series can fail, and most failures start quietly: a GPU throwing memory errors, a collective that never completes, a configuration change that loads a control plane, a model that starts giving slightly worse answers. Site reliability engineers build the telemetry that sees these problems, the alerts that wake someone up, and the incident process that fixes them and makes sure they do not happen again. This part covers that layer, with two unusually candid public postmortems from OpenAI and Anthropic as case studies.

FL
FrontierAGI Team
โš ๏ธ How this was researched Case studies come from OpenAI's public write-up of its 11 December 2024 outage and Anthropic's September 2025 engineering postmortem, plus press coverage; a few details found only in secondary coverage are marked โš ๏ธ secondary. Tooling facts come from NVIDIA and PyTorch documentation; role descriptions from real SRE postings (mostly NVIDIA; no OpenAI or Anthropic SRE postings were found). Fleet health, lemon nodes and goodput are covered in Producing Intelligence Part 9, and a research engineer's on-call life in The On-Call Reality. Calculators and demos are illustrations. See Reading the numbers.

1. Where we are in the stack

The AI factory stack, by role

2. The roles: SREs for training clusters and for inference

NVIDIA's postings for a Senior Site Reliability Engineer, AI Research Clusters ask for experience with AI/HPC job schedulers (Slurm preferred), and a related posting asks for operational experience on clusters of at least 5,000 GPUs. NVIDIA also advertises a Senior SRE, GPU Clusters; Genmo's SRE role centres on Kubernetes operations and GPU scheduling, and Boson AI hires SREs for its GPU clusters. In practice the job splits between keeping training clusters productive and keeping inference services fast and available.

Two flavours of AI SRE

Our reading of postings (1โ€“5). The day timelines are composite illustrations.

3. What's different about observing AI

Classic observability rests on metrics, logs and traces. AI factories add signals that ordinary web services never needed. Click one.

Four kinds of signal

4. What lives at this layer

The observability stack

5. SLOs and error budgets

Reliability starts with a number everyone agrees on. A service-level objective says, for example, that 99.9% of requests must succeed within a latency target over a month. The gap to 100% is the error budget: the amount of failure the service can afford. Spend it on a bad incident and risky launches pause until it recovers; leave it unspent and the team is probably moving too cautiously.

Error-budget calculator

6. Case study: OpenAI's 11 December 2024 outage

On 11 December 2024, ChatGPT, the API and Sora were down for over four hours, from 3:16 PM to 7:38 PM Pacific time. OpenAI's public write-up traced it to its own observability work. A new telemetry service, meant to improve visibility into its Kubernetes clusters, was deployed with a configuration that triggered resource-intensive calls to the Kubernetes API. That overwhelmed the API servers and took down the control plane in most large clusters. The control plane also supported DNS for many services, and because DNS answers were cached, failures appeared gradually, after the change had already reached every cluster. With the control plane down, engineers could not easily reach the clusters to roll back, which lengthened the recovery. The change had been merged at 2:23 PM and applied to all clusters between 2:51 and 3:20 PM.

Anatomy of the outage

Times as published by OpenAI (Pacific time). Step through.

๐Ÿ’ก Lessons OpenAI drew Test changes for their effect on the control plane, not just the service; roll out gradually with automatic halts; keep a "break-glass" path into clusters that does not depend on the systems that might fail; and decouple critical dependencies such as DNS from the Kubernetes control plane.

7. Case study: Anthropic's September 2025 postmortem

The second case is subtler. Between August and early September 2025, three separate infrastructure bugs intermittently degraded the quality of Claude's responses. Anthropic's postmortem stressed that it never reduces model quality because of demand, time of day or server load; the problems users noticed came from bugs. One bug misrouted some requests to servers configured for a different context window, starting on 5 August; secondary coverage reports it peaked at 16% of Sonnet 4 requests in the worst hour on 31 August โš ๏ธ secondary. Others produced wrong or corrupted output tokens, including issues in the TPU serving stack and its compiler โš ๏ธ secondary detail. Because each bug affected different platforms in different ways, and because Anthropic's privacy practices limit engineers' access to user conversations that were not reported as feedback, the problems were hard to detect and reproduce. Anthropic's fixes included evaluations that can better distinguish normal from faulty behaviour and continuous quality monitoring on production systems.

The silent failure

Illustrative. Latency and error dashboards stay green while output quality on a slice of traffic drifts. Toggle a production quality monitor.

8. Case study: Meta's flight recorder

For training clusters, the most dreaded alert is an NCCL watchdog timeout: a collective operation that never finishes, with an error message that says little about why. A PyTorch blog post describes the Flight Recorder, which keeps a ring buffer of recent collective operations on every rank and dumps it when a job hangs, so engineers can line up what each rank was doing and find the one that diverged; the post includes how it is used inside Meta, and NVIDIA's resiliency tooling offers analysis that returns the "missing ranks" as root-cause suspects. Combined with the health checks and lemon-node detection from Producing Intelligence Part 9, it is how a 54-day run with 466 interruptions still achieved more than 90% effective training time.

9. Incident response and postmortems

When something breaks, process matters as much as skill: a clear severity level, one incident commander coordinating, someone communicating with users, and a blameless postmortem afterwards that fixes systems rather than blaming people. Both OpenAI and Anthropic published theirs, which is unusual and valuable for the whole industry. Try handling one.

You're on call

An inference latency alert fires at 02:10. Choose your moves.

10. Handoffs and careers

Who SREs work with

๐Ÿงญ Getting into the role
Typical backgrounds
SREs and production engineers from large web companies, HPC operations staff, infrastructure software engineers.
Core skills
Linux and Kubernetes or Slurm, Prometheus-style metrics and alerting, logs and tracing, GPU telemetry (DCGM), incident command, writing postmortems, automation, calm under pressure.
Growth path
SRE โ†’ owner of a cluster's or service's reliability โ†’ reliability lead across the factory.
Further reading
The On-Call Reality ยท Keeping the Factory Running ยท Part 10: Inference

๐Ÿ—‚๏ธ Case file

OpenAI, Dec 2024
New telemetry service overloaded Kubernetes control planes; ~4h20m outage; DNS caching hid the problem; lockout slowed rollback.
Anthropic, Augโ€“Sep 2025
Three infrastructure bugs degraded output quality intermittently; hard to detect; fixed with better evals and continuous production quality monitoring.
Meta
Flight Recorder for NCCL hangs; health checks and lemon detection; >90% effective training time for Llama 3 405B.
Verdict AI reliability has two halves: classic SRE (keep the systems up, roll out carefully, keep a way in when things break) and something new (notice when a model is up but wrong). The two public postmortems show both: an observability tool that caused an outage, and quality bugs that standard dashboards could not see. The SREs who combine infrastructure discipline with model-quality monitoring are what make AI services trustworthy.

๐Ÿ‘€ Next in the series

Part 12: Security and access โ€” the engineers who protect model weights, isolate tenants and secure the AI supply chain.

๐Ÿ”ข Reading the numbers

๐Ÿ“š References