Home›Blog›Frontier Lab Engineering — Debugging Distributed Jobs
Frontier Lab Engineering Practicum · Article 3 of 9🧯
Debugging a Distributed Job That Won't Converge (Or Won't Start)
A practical runbook for the two failure categories every research engineer eventually meets: a job that never gets off the ground, and a job that runs but produces garbage. Both fail silently by default — this article covers how to triage which category you're in, and the specific, real causes behind each.
Neural Net Training Fails Silently — Distributed Training Fails Silently at Scale
Andrej Karpathy's widely-read "A Recipe for Training Neural Networks" opens with a blunt warning: "neural net training fails silently." A misconfigured model doesn't usually crash — it trains, produces numbers, and just quietly doesn't work. Distributed training (Article 10 of the Foundations series) adds an entire second layer of silent failure on top of that: a bug in how data or gradients are synchronized across GPUs can produce a job that runs, logs a plausible-looking loss curve, and is nonetheless wrong in a way single-GPU debugging instincts won't catch.
This article is organized around the first question you should ask when a job is behaving badly: did it fail to start, or is it running but not doing the right thing? These have almost entirely different root causes and almost entirely different debugging approaches, and conflating them wastes real time.
2Failure categories this article separates: won't start, and won't converge
~35Manual restarts in Meta's OPT-175B logbook — recategorized in Part 5
1 GPUWhere a good distributed bug reproduction should try to shrink to first, before debugging at full scale
"Silently"The one word that appears in both Karpathy's training advice and this article's title, and isn't a coincidence
Part 1 — Triage: Which Failure Are You Actually Looking At?
The First, Most Important Diagnostic Question
The single question that splits nearly every debugging session in two: has the step counter and loss ever moved at all?
Part 2 — Won't Start
The Job Never Even Gets Off the Ground
If nothing has logged even a single training step, the problem is almost always infrastructure or configuration, not the model or the math.
1
Environment mismatch — a library version difference between the machine a config was written on and the cluster it's launched on (Article 1 of this series' container-image discipline exists specifically to prevent this).
2
Resource allocation failures — requesting more GPUs, memory, or a specific interconnect topology than the scheduler (Article 10 of the Foundations series) currently has available, often failing with a queue error rather than a training error.
3
NCCL initialization hangs — the collective-communication library underneath data and tensor parallelism (Article 10's ring-allreduce) failing to establish connections between GPUs, often due to network topology misconfiguration or firewall rules; PyTorch's own distributed debugging documentation specifically recommends setting TORCH_DISTRIBUTED_DEBUG=DETAIL to surface these hangs with useful logging instead of an opaque freeze.
4
Checkpoint loading errors — resuming from a checkpoint saved under a different parallelism configuration (Article 10's tensor/pipeline sharding) than the current run is using, a mismatch that often fails cryptically rather than with a clear "shapes don't match" message.
Part 3 — Won't Converge
The Job Runs, But the Numbers Are Wrong
Once you know steps are actually advancing, the debugging shifts entirely to the training signal itself.
Symptom
Likely Cause
Where to Look First
Loss becomes NaN/Inf suddenly
Exploding gradients (Article 4 of the Foundations series)
Gradient clipping settings, learning rate, mixed-precision numerical range
Loss plateaus immediately, never drops
Learning rate far too low, or a frozen/misconnected parameter
Optimizer config, verify gradients are actually flowing to every parameter
Loss looks fine but eval metrics are terrible
Data pipeline bug (Article 1 of this series' Scenario A) — training on the wrong or corrupted data
Manually inspect a batch of actual training examples, not just the loss number
Loss curve looks fine in aggregate, but results don't reproduce a known baseline
Silent distributed-specific bug — see Part 4
Per-GPU/per-rank logging, not just the aggregated loss
Part 4 — Silent Distributed-Specific Bugs
The Bugs That Only Exist Because You're Distributed
These are the failure modes with no single-GPU equivalent — the reason distributed training debugging is its own discipline, not just "the same debugging, but bigger."
1
Every GPU seeing the same data shard. A bug in how a distributed data sampler assigns shards (Article 10's data parallelism) can cause every GPU to train on an identical subset of data instead of complementary ones — the aggregate loss curve can still look completely plausible, because each GPU is training correctly on its (wrongly identical) data, while the model never actually sees the full dataset's diversity.
2
Mismatched precision across ranks. If some GPUs are running in a different numerical precision (FP16 vs. BF16 vs. FP32) than others due to a config or hardware inconsistency, gradient synchronization (Article 10's ring-allreduce) can silently average numerically incompatible values, producing subtly wrong updates that don't crash anything.
3
Stale weights after a partial restart. If a checkpoint restore only partially succeeds — some ranks load the new checkpoint, others silently keep old in-memory weights — the resulting run trains, logs a loss, and is quietly comparing apples to oranges from that point forward.
A distributed job's aggregate loss curve can look completely healthy while several of these bugs are actively happening — the aggregation itself is what hides the problem, which is exactly why per-rank, not just aggregate, logging matters.
Part 5 — The OPT-175B Logbook, Recategorized
Applying This Article's Framework to a Real Incident Record
Meta's public OPT-175B training logbook (cited throughout this site's Foundations series) is the best real, granular record available of what these failures actually look like at scale. Read through this article's lens, its roughly 35 manual restarts split cleanly into this article's two categories: hardware failures and NCCL communication errors are "won't start" (or "stopped running entirely") problems — Part 2's territory — while loss divergences and instability required "won't converge" diagnosis — Part 3's territory, often resolved by restarting from an earlier checkpoint with adjusted hyperparameters, precisely the gradient-instability response Part 3's table recommends.
Part 6 — The Debugging Toolkit
Practical Habits That Make All of the Above Findable
1
Shrink to a minimal reproduction. Before debugging at full cluster scale, try to reproduce the symptom on the smallest possible setup — ideally a single GPU or a handful — using Article 2 of this series' config-override discipline to change only the scale, not the logic.
2
Log per-rank, not just aggregate, metrics. Directly addresses Part 4's silent-bug class — an aggregate loss can hide a problem that's obvious the moment you compare individual GPUs' local losses against each other.
3
Bisect by commit or config change, not by guessing. If a run was healthy last week and isn't now, treat it like any other regression: find the last known-good config/commit and the first known-bad one, then narrow the gap systematically rather than jumping to a hypothesis first.
Part 7 — Real Scenarios
Real Scenario Walkthroughs
🪞Scenario A — The Suspiciously Perfect Loss Curve
A team notices their distributed run's loss curve is unusually smooth compared to a known single-GPU baseline of the same model — smoother than expected rather than worse. Applying Part 4's framework, this is a legitimate reason for suspicion, not celebration: it's a classic symptom of every GPU training on identical, over-duplicated data rather than the intended diverse shards, artificially reducing the noise a real, correctly-sharded batch would show.
The lesson: an unexpectedly good-looking metric deserves the same scrutiny as a bad one — "too smooth" is itself a diagnosable symptom once you know what causes it.
🧊Scenario B — A Job That Hangs With No Error Message
One of the most frustrating "won't start" patterns: the job doesn't crash, doesn't log an error, and simply never progresses past initialization. Per Part 2, this is a strong signal to check NCCL initialization specifically, since a communication-layer hang between GPUs often produces exactly this silent-freeze behavior rather than a clean failure — enabling detailed distributed debugging logging (as PyTorch's own docs recommend) is usually the fastest path to an actual error message instead of just a hang.
The lesson: "hangs with no error" is itself diagnostic information pointing toward the communication layer, not a random, unknowable failure.
Readiness Checklist
1
Given a job with no output at all, can you name at least three distinct categories of cause to check, in order?
2
Can you distinguish a NaN-loss bug (Article 4's exploding gradients) from a data-pipeline bug (Article 1 of this series' Scenario A) by their symptoms alone?
3
Can you explain how every GPU seeing identical data could produce a plausible-looking aggregate loss curve?
4
Have you ever shrunk a distributed bug down to a single-GPU reproduction before debugging it at full scale?
⚠️ What's Missing or Uncertain
This article is a triage framework, not an exhaustive troubleshooting manual. Specific error messages and their exact causes vary significantly across cluster hardware, scheduler software, and framework versions — treat the categories above (won't start vs. won't converge, and the silent-bug list in Part 4) as the mental model to bring to a real incident, not a literal lookup table for every possible error string.
Where This Series Goes Next
Article 4 moves from debugging a running job to the economics surrounding it: how compute is actually requested, scheduled, and priced day to day — job scheduler queues, spot vs. reserved capacity, and what "utilization" means to the person approving your compute budget, extending Article 10 of the Foundations series' systems theory into the practical, organizational reality of getting GPU time in the first place.
🧭 Closing — Two Categories, and a Habit of Suspicion
🎯 The Bottom Line
Nearly every distributed training failure sorts cleanly into "never started" (infrastructure, environment, communication setup) or "runs but wrong" (gradients, data, or a distributed-specific silent bug) — and knowing which category you're in before you start guessing saves real time. The deeper habit this article is really teaching is treating a suspiciously good-looking metric with the same scrutiny as a bad one, since several of this article's most insidious bugs (duplicated data shards, mismatched precision) hide behind a perfectly plausible aggregate loss curve. The OPT-175B logbook's ~35 restarts are proof this isn't a rare inconvenience — it's the actual, well-documented shape of the job.