Who Watches the Factory? Observability, Incidents and the SREs Behind AI Services
Every layer in this series can fail, and most failures start quietly: a GPU throwing memory errors, a collective that never completes, a configuration change that loads a control plane, a model that starts giving slightly worse answers. Site reliability engineers build the telemetry that sees these problems, the alerts that wake someone up, and the incident process that fixes them and makes sure they do not happen again. This part covers that layer, with two unusually candid public postmortems from OpenAI and Anthropic as case studies.
1. Where we are in the stack
The AI factory stack, by role
2. The roles: SREs for training clusters and for inference
NVIDIA's postings for a Senior Site Reliability Engineer, AI Research Clusters ask for experience with AI/HPC job schedulers (Slurm preferred), and a related posting asks for operational experience on clusters of at least 5,000 GPUs. NVIDIA also advertises a Senior SRE, GPU Clusters; Genmo's SRE role centres on Kubernetes operations and GPU scheduling, and Boson AI hires SREs for its GPU clusters. In practice the job splits between keeping training clusters productive and keeping inference services fast and available.
Two flavours of AI SRE
Our reading of postings (1โ5). The day timelines are composite illustrations.
3. What's different about observing AI
Classic observability rests on metrics, logs and traces. AI factories add signals that ordinary web services never needed. Click one.
Four kinds of signal
4. What lives at this layer
The observability stack
5. SLOs and error budgets
Reliability starts with a number everyone agrees on. A service-level objective says, for example, that 99.9% of requests must succeed within a latency target over a month. The gap to 100% is the error budget: the amount of failure the service can afford. Spend it on a bad incident and risky launches pause until it recovers; leave it unspent and the team is probably moving too cautiously.
Error-budget calculator
6. Case study: OpenAI's 11 December 2024 outage
On 11 December 2024, ChatGPT, the API and Sora were down for over four hours, from 3:16 PM to 7:38 PM Pacific time. OpenAI's public write-up traced it to its own observability work. A new telemetry service, meant to improve visibility into its Kubernetes clusters, was deployed with a configuration that triggered resource-intensive calls to the Kubernetes API. That overwhelmed the API servers and took down the control plane in most large clusters. The control plane also supported DNS for many services, and because DNS answers were cached, failures appeared gradually, after the change had already reached every cluster. With the control plane down, engineers could not easily reach the clusters to roll back, which lengthened the recovery. The change had been merged at 2:23 PM and applied to all clusters between 2:51 and 3:20 PM.
Anatomy of the outage
Times as published by OpenAI (Pacific time). Step through.
7. Case study: Anthropic's September 2025 postmortem
The second case is subtler. Between August and early September 2025, three separate infrastructure bugs intermittently degraded the quality of Claude's responses. Anthropic's postmortem stressed that it never reduces model quality because of demand, time of day or server load; the problems users noticed came from bugs. One bug misrouted some requests to servers configured for a different context window, starting on 5 August; secondary coverage reports it peaked at 16% of Sonnet 4 requests in the worst hour on 31 August โ ๏ธ secondary. Others produced wrong or corrupted output tokens, including issues in the TPU serving stack and its compiler โ ๏ธ secondary detail. Because each bug affected different platforms in different ways, and because Anthropic's privacy practices limit engineers' access to user conversations that were not reported as feedback, the problems were hard to detect and reproduce. Anthropic's fixes included evaluations that can better distinguish normal from faulty behaviour and continuous quality monitoring on production systems.
The silent failure
Illustrative. Latency and error dashboards stay green while output quality on a slice of traffic drifts. Toggle a production quality monitor.
8. Case study: Meta's flight recorder
For training clusters, the most dreaded alert is an NCCL watchdog timeout: a collective operation that never finishes, with an error message that says little about why. A PyTorch blog post describes the Flight Recorder, which keeps a ring buffer of recent collective operations on every rank and dumps it when a job hangs, so engineers can line up what each rank was doing and find the one that diverged; the post includes how it is used inside Meta, and NVIDIA's resiliency tooling offers analysis that returns the "missing ranks" as root-cause suspects. Combined with the health checks and lemon-node detection from Producing Intelligence Part 9, it is how a 54-day run with 466 interruptions still achieved more than 90% effective training time.
9. Incident response and postmortems
When something breaks, process matters as much as skill: a clear severity level, one incident commander coordinating, someone communicating with users, and a blameless postmortem afterwards that fixes systems rather than blaming people. Both OpenAI and Anthropic published theirs, which is unusual and valuable for the whole industry. Try handling one.
You're on call
An inference latency alert fires at 02:10. Choose your moves.
10. Handoffs and careers
Who SREs work with
- Typical backgrounds
- SREs and production engineers from large web companies, HPC operations staff, infrastructure software engineers.
- Core skills
- Linux and Kubernetes or Slurm, Prometheus-style metrics and alerting, logs and tracing, GPU telemetry (DCGM), incident command, writing postmortems, automation, calm under pressure.
- Growth path
- SRE โ owner of a cluster's or service's reliability โ reliability lead across the factory.
- Further reading
- The On-Call Reality ยท Keeping the Factory Running ยท Part 10: Inference
๐๏ธ Case file
- OpenAI, Dec 2024
- New telemetry service overloaded Kubernetes control planes; ~4h20m outage; DNS caching hid the problem; lockout slowed rollback.
- Anthropic, AugโSep 2025
- Three infrastructure bugs degraded output quality intermittently; hard to detect; fixed with better evals and continuous production quality monitoring.
- Meta
- Flight Recorder for NCCL hangs; health checks and lemon detection; >90% effective training time for Llama 3 405B.
๐ Next in the series
Part 12: Security and access โ the engineers who protect model weights, isolate tenants and secure the AI supply chain.
๐ข Reading the numbers
- OpenAI outage times, cause and contributing factors come from OpenAI's status-page write-up and TechCrunch's coverage.
- Anthropic's three bugs, the no-degradation statement, detection challenges and fixes come from its postmortem as summarised by Gigazine, Simon Willison and InfoQ; the 16% peak and TPU/compiler specifics are โ ๏ธ secondary.
- Flight Recorder and DCGM exporter facts come from PyTorch and NVIDIA documentation.
- Role descriptions come from NVIDIA, Genmo and Boson AI postings.
- The error-budget calculator, silent-failure chart and on-call role-play are illustrations.
๐ References
- Postmortems
- OpenAI: API, ChatGPT & Sora facing issues โ incident write-up (December 2024)
- TechCrunch: OpenAI blames its massive ChatGPT outage on a new telemetry service
- Anthropic: A postmortem of three recent issues (September 2025)
- Simon Willison on Anthropic's postmortem ยท InfoQ coverage
- Tools
- PyTorch blog: Flight Recorder, a new lens for NCCL watchdog timeouts
- NVIDIA: dcgm-exporter
- Roles
- NVIDIA: Senior SRE, AI Research Clusters (posting)
- NVIDIA: Senior SRE, GPU Clusters (posting)
- Related on this site
- The On-Call Reality ยท Keeping the Factory Running ยท Part 10: Inference platform