The Same Machine, 100,000 Times: The Engineers Who Keep Kernels, Drivers and CUDA in Step
Once bring-up hands over a rack (Part 1), it joins a fleet where every machine must run the right Linux kernel, the right NVIDIA driver and a compatible CUDA stack, and keep doing so while all of those change underneath it. A training job spread over thousands of servers fails if even a few disagree. This part follows the systems software engineers who own that layer: the operating system, drivers, images and provisioning that make 100,000 machines behave like one.
1. Where we are in the stack
The AI factory stack, by role
2. The role: systems software engineer for the fleet
OpenAI's posting for a Software Engineer, Compute Foundations Systems describes the job well. The team sits in Frontier Systems Foundations and owns the operating-system layer of frontier training clusters: kernels, drivers, OS images, provisioning and firmware integration. It prefers people who have built bare-metal provisioning or OS-reprovisioning tooling, worked with GPU systems, HPC clusters or other accelerators, and contributed to Linux, kernel modules or distributions. On the vendor side, NVIDIA employs CUDA driver engineers who write the kernel-mode GPU drivers that these teams deploy.
What the role needs
Skill emphasis distilled from the postings (our judgement, 1โ5). The day timeline is a composite illustration.
3. What lives at this layer
Between the firmware from Part 1 and a researcher's PyTorch code sit several pieces of software that must all agree. Click a layer.
The runtime layer cake
On Kubernetes clusters, NVIDIA's GPU Operator automates much of this: it deploys the driver as a container, the Kubernetes device plugin, the NVIDIA Container Toolkit, automatic node labelling (GPU Feature Discovery) and DCGM-based monitoring. Because the driver depends on both the OS and the kernel version, the operator still has to match them. Slurm-based HPC clusters do the same work with images and configuration management instead.
4. The compatibility puzzle
The central problem of this layer is that its pieces update on different clocks. Kernels get security fixes, NVIDIA ships new driver branches, researchers want the newest CUDA and PyTorch, and firmware changes arrive from Part 1. Meta's approach, described in its 2024 post on maintaining AI capacity, splits the stack in two:
- Inside the job's container: the CUDA user-space libraries ship with the job and are identical across a training job, so every rank runs exactly the same code.
- Below the container: the kernel, drivers and firmware are upgraded gradually across the fleet, and each change must be compatible with both the previous and the new version, so a job can run on a mix of hosts while a rollout is half done.
NVIDIA's forward compatibility package (named cuda-compat-<major>-<minor>) helps: on data-centre GPUs it lets a newer CUDA toolkit run on an older kernel-mode driver, so containers can move ahead of the fleet. It has limits: some CUDA features need a newer kernel-mode driver to work. And since the R560 release in 2024, NVIDIA's open-source GPU kernel modules are the default and recommended choice, for Turing-generation GPUs and newer.
Will this job run?
Illustrative compatibility rules (version numbers simplified). Pick the fleet's driver and the job's CUDA version.
5. Rolling it out: Meta's maintenance trains
Meta keeps its AI clusters current with maintenance trains: on a regular cadence, a small number of hosts are taken out of production, given every applicable upgrade at once and returned. Host-level components are the hard part, because they can take hours to install or need a reboot; longer upgrades may be spread over several train cycles. Because Meta can't take large amounts of capacity offline (many jobs are time-critical or always on), the trains keep each slice small and rely on the compatibility rule above.
Meta's kernel team adds that the company tries to run the newest kernel of any hyperscaler, which means it often finds kernel problems before others do; for urgent security fixes it has used kernel live patching to update servers without a reboot โ ๏ธ secondary.
A maintenance train
Illustrative: 64 hosts, a few at a time drained, upgraded and returned. Press play.
6. When it breaks: Xid errors and triage
When something goes wrong at this layer, the first clue is usually an Xid: an error report that the NVIDIA driver writes to the kernel log. NVIDIA is clear that an Xid is a starting point, not a diagnosis: the same code can come from hardware, the driver or the application. NVSwitch systems add SXid messages from the switches (Hopper and earlier), and a fault in Fabric Manager, the service that configures NVSwitch, can look like a GPU fault. NVIDIA's triage guidance is to collect driver and OS versions, the key log lines, nvidia-bug-report.sh output, Fabric Manager logs and DCGM diagnostics.
Xid detective
Pick a symptom. Simplified from NVIDIA's documentation and vendor runbooks; real triage needs the full logs.
7. Images, provisioning and reproducibility
Every host boots from a known image: a tested combination of OS, kernel, driver and tools. Bare-metal provisioning systems install or reinstall that image automatically, which is also the fastest way to fix a host whose software has drifted. On top, jobs run in containers that pin CUDA, cuDNN, NCCL and PyTorch versions, often starting from vendor base images such as NVIDIA's NGC containers. Pinning matters beyond convenience: when a run stops converging, the first question is whether anything changed, and reproducible images make that answerable (see Debugging a Distributed Job).
8. Handoffs and careers
Who this team works with
- Typical backgrounds
- Linux kernel and distribution engineers, HPC system administrators, embedded and driver developers, production engineers at large web companies.
- Core skills
- Linux internals, kernel modules, package and image building, provisioning (PXE, image pipelines), containers, NVIDIA driver and CUDA tooling, careful rollout and rollback practice.
- Growth path
- Fleet systems engineer โ owner of OS images and rollouts โ staff engineer for the platform, or a move into the vendor side (driver development) or cluster platform work.
- Further reading
- Part 1: Bring-up ยท Keeping the Factory Running ยท Day One at a Frontier Lab
๐๏ธ Case file: Meta
- Rule
- CUDA ships in the job's container and is homogeneous across a job; lower layers roll out with old/new compatibility.
- Mechanism
- Maintenance trains: small slices of hosts drained, fully upgraded and returned on a cadence.
- Kernel
- Runs the newest kernel of the hyperscalers; live patching for urgent fixes โ ๏ธ secondary.
- Scale (2026)
- Fixes to 100K+ GPUs in hours; P90 pod upgrade ~2 h; MTBI >54K GPU-hours โ ๏ธ secondary.
๐ Next in the series
Part 3: Networking software โ the fabric engineers who configure RoCE and InfiniBand, tune congestion control and NCCL, and decide where jobs land on the network.
๐ข Reading the numbers
- Driver defaults, forward compatibility, GPU Operator components and Xid guidance come from NVIDIA's documentation.
- Maintenance trains and the container/host split come from Meta's June 2024 post, read through Gigazine's summary and the post's figure captions.
- Newest-kernel policy comes from a Meta kernel talk at Linux Plumbers 2025; live patching from secondary coverage โ ๏ธ.
- Prometheus figures come from StorageReview and other outlets โ ๏ธ secondary.
- Role description comes from OpenAI's job posting.
- Skills radar, day timeline, compatibility rules, maintenance-train animation and Xid quiz are illustrations.
๐ References
- NVIDIA
- NVIDIA driver installation guide: kernel modules (open modules default from R560)
- NVIDIA: CUDA Compatibility ยท Forward compatibility
- NVIDIA GPU Operator overview
- NVIDIA: Xid errors ยท GPU node triage
- Meta
- Meta Engineering: Maintaining large-scale AI capacity at Meta (2024)
- Gigazine: How does Meta maintain the infrastructure to run AI at scale?
- Linux Plumbers 2025: kernel monitoring on Meta's fleet
- StorageReview: Meta Prometheus gigawatt supercluster (2026)
- Roles
- OpenAI: Software Engineer, Compute Foundations Systems (posting)
- NVIDIA: System Software Engineer, CUDA Driver (posting)
- Related on this site
- Part 1: Day Zero ยท Producing Intelligence Part 1: The Software Stack ยท Debugging a Distributed Job