Home โ€บ Blog โ€บ OS, Drivers and Runtime
Who Runs the AI Factory ยท Part 2 ยท OS, drivers & runtime

The Same Machine, 100,000 Times: The Engineers Who Keep Kernels, Drivers and CUDA in Step

Once bring-up hands over a rack (Part 1), it joins a fleet where every machine must run the right Linux kernel, the right NVIDIA driver and a compatible CUDA stack, and keep doing so while all of those change underneath it. A training job spread over thousands of servers fails if even a few disagree. This part follows the systems software engineers who own that layer: the operating system, drivers, images and provisioning that make 100,000 machines behave like one.

FL
FrontierAGI Team
โš ๏ธ How this was researched Sources are NVIDIA's documentation (driver installation, CUDA compatibility, GPU Operator, Xid errors), Meta's June 2024 engineering post on maintaining AI capacity (which we read through a published summary and the post's figure captions), Meta's kernel team talks, coverage of Meta's Prometheus cluster (October 2026, figures reported by secondary outlets and marked โš ๏ธ secondary) and real job postings. The compatibility matrix, maintenance-train animation and Xid quiz are illustrations. See Reading the numbers.

1. Where we are in the stack

The AI factory stack, by role

2. The role: systems software engineer for the fleet

OpenAI's posting for a Software Engineer, Compute Foundations Systems describes the job well. The team sits in Frontier Systems Foundations and owns the operating-system layer of frontier training clusters: kernels, drivers, OS images, provisioning and firmware integration. It prefers people who have built bare-metal provisioning or OS-reprovisioning tooling, worked with GPU systems, HPC clusters or other accelerators, and contributed to Linux, kernel modules or distributions. On the vendor side, NVIDIA employs CUDA driver engineers who write the kernel-mode GPU drivers that these teams deploy.

What the role needs

Skill emphasis distilled from the postings (our judgement, 1โ€“5). The day timeline is a composite illustration.

3. What lives at this layer

Between the firmware from Part 1 and a researcher's PyTorch code sit several pieces of software that must all agree. Click a layer.

The runtime layer cake

On Kubernetes clusters, NVIDIA's GPU Operator automates much of this: it deploys the driver as a container, the Kubernetes device plugin, the NVIDIA Container Toolkit, automatic node labelling (GPU Feature Discovery) and DCGM-based monitoring. Because the driver depends on both the OS and the kernel version, the operator still has to match them. Slurm-based HPC clusters do the same work with images and configuration management instead.

4. The compatibility puzzle

The central problem of this layer is that its pieces update on different clocks. Kernels get security fixes, NVIDIA ships new driver branches, researchers want the newest CUDA and PyTorch, and firmware changes arrive from Part 1. Meta's approach, described in its 2024 post on maintaining AI capacity, splits the stack in two:

NVIDIA's forward compatibility package (named cuda-compat-<major>-<minor>) helps: on data-centre GPUs it lets a newer CUDA toolkit run on an older kernel-mode driver, so containers can move ahead of the fleet. It has limits: some CUDA features need a newer kernel-mode driver to work. And since the R560 release in 2024, NVIDIA's open-source GPU kernel modules are the default and recommended choice, for Turing-generation GPUs and newer.

Will this job run?

Illustrative compatibility rules (version numbers simplified). Pick the fleet's driver and the job's CUDA version.

5. Rolling it out: Meta's maintenance trains

Meta keeps its AI clusters current with maintenance trains: on a regular cadence, a small number of hosts are taken out of production, given every applicable upgrade at once and returned. Host-level components are the hard part, because they can take hours to install or need a reboot; longer upgrades may be spread over several train cycles. Because Meta can't take large amounts of capacity offline (many jobs are time-critical or always on), the trains keep each slice small and rely on the compatibility rule above.

Meta's kernel team adds that the company tries to run the newest kernel of any hyperscaler, which means it often finds kernel problems before others do; for urgent security fixes it has used kernel live patching to update servers without a reboot โš ๏ธ secondary.

A maintenance train

Illustrative: 64 hosts, a few at a time drained, upgraded and returned. Press play.

๐Ÿ“ˆ At Prometheus scale (October 2026) โš ๏ธ secondary Coverage of Meta's Prometheus supercluster reports that Meta's automation can push driver and firmware fixes to more than 100,000 GPUs within a few hours, that the P90 time to upgrade one GB200 pod fell from over seven hours to about two, and that firmware fixes raised the mean time between interruptions from about 47,000 to more than 54,000 GPU-hours, against a goal of roughly 100,000. These figures come from StorageReview and other outlets reporting Meta's statements; we have not read Meta's primary post.

6. When it breaks: Xid errors and triage

When something goes wrong at this layer, the first clue is usually an Xid: an error report that the NVIDIA driver writes to the kernel log. NVIDIA is clear that an Xid is a starting point, not a diagnosis: the same code can come from hardware, the driver or the application. NVSwitch systems add SXid messages from the switches (Hopper and earlier), and a fault in Fabric Manager, the service that configures NVSwitch, can look like a GPU fault. NVIDIA's triage guidance is to collect driver and OS versions, the key log lines, nvidia-bug-report.sh output, Fabric Manager logs and DCGM diagnostics.

Xid detective

Pick a symptom. Simplified from NVIDIA's documentation and vendor runbooks; real triage needs the full logs.

7. Images, provisioning and reproducibility

Every host boots from a known image: a tested combination of OS, kernel, driver and tools. Bare-metal provisioning systems install or reinstall that image automatically, which is also the fastest way to fix a host whose software has drifted. On top, jobs run in containers that pin CUDA, cuDNN, NCCL and PyTorch versions, often starting from vendor base images such as NVIDIA's NGC containers. Pinning matters beyond convenience: when a run stops converging, the first question is whether anything changed, and reproducible images make that answerable (see Debugging a Distributed Job).

8. Handoffs and careers

Who this team works with

๐Ÿงญ Getting into the role
Typical backgrounds
Linux kernel and distribution engineers, HPC system administrators, embedded and driver developers, production engineers at large web companies.
Core skills
Linux internals, kernel modules, package and image building, provisioning (PXE, image pipelines), containers, NVIDIA driver and CUDA tooling, careful rollout and rollback practice.
Growth path
Fleet systems engineer โ†’ owner of OS images and rollouts โ†’ staff engineer for the platform, or a move into the vendor side (driver development) or cluster platform work.
Further reading
Part 1: Bring-up ยท Keeping the Factory Running ยท Day One at a Frontier Lab

๐Ÿ—‚๏ธ Case file: Meta

Rule
CUDA ships in the job's container and is homogeneous across a job; lower layers roll out with old/new compatibility.
Mechanism
Maintenance trains: small slices of hosts drained, fully upgraded and returned on a cadence.
Kernel
Runs the newest kernel of the hyperscalers; live patching for urgent fixes โš ๏ธ secondary.
Scale (2026)
Fixes to 100K+ GPUs in hours; P90 pod upgrade ~2 h; MTBI >54K GPU-hours โš ๏ธ secondary.
Verdict This layer is about discipline more than invention: deciding what must be identical inside a job, what may differ underneath, and how to change 100,000 machines without anyone noticing. Meta's split, with CUDA pinned in the container and everything below it rolled out compatibly in small trains, is a clear template. As fleets grow and rack-scale systems make each change riskier, the speed of safe rollout has become a reliability metric in its own right.

๐Ÿ‘€ Next in the series

Part 3: Networking software โ€” the fabric engineers who configure RoCE and InfiniBand, tune congestion control and NCCL, and decide where jobs land on the network.

๐Ÿ”ข Reading the numbers

๐Ÿ“š References