Who Gets the GPUs? The Engineers Who Schedule an AI Factory
An AI factory never has enough GPUs. A frontier pre-training run wants thousands of them for months, researchers want a few hundred for an afternoon, inference wants guaranteed capacity, and evaluation wants everything right before a launch. Someone has to decide who runs where and when, and do it so that expensive hardware is never idle. This part follows the cluster platform engineers who build and run the schedulers, from Slurm and Kubernetes to Meta's global scheduler, MAST.
1. Where we are in the stack
The AI factory stack, by role
2. The role: cluster platform engineer
Postings for this role blend HPC and cloud-native skills. Perplexity's Member of Technical Staff, GPU Cluster Infrastructure asks for deep Kubernetes expertise, including custom operators and multi-cluster management, to build a self-serve compute platform for inference engineers and researchers. A staff HPC posting centres on Slurm cluster architecture and multi-tenant scheduling policies, including modelling the physical network fabric for GPU scheduling and running Slurm on Kubernetes with Slinky. Crusoe's Staff Software Engineer, Slurm lists partitions, QoS, preemption, accounting and container integration. Typical asks are 5โ8+ years in infrastructure with substantial GPU, HPC or ML cluster experience, plus familiarity with NCCL, RDMA networking and GPU driver stacks.
What the role needs
Our reading of the postings (1โ5). The day timeline is a composite illustration.
3. The scheduling problem
A GPU scheduler has to satisfy requirements that pull against each other:
- Gang scheduling. A distributed training job needs all its GPUs at once. Giving it 900 of 1,024 is worse than nothing: they sit idle waiting for the rest.
- Priority and preemption. A launch-critical run must be able to push lower-priority work off the machines, which must then checkpoint and requeue.
- Fair share. Teams have quotas; idle quota should be lent out, then reclaimed.
- Topology. A job's GPUs should be close on the network. Llama 3's cluster grouped GPUs into 3,072-GPU pods with 1:7 oversubscription between pods (see Part 3), so a job split across pods runs slower.
- Health. Never hand out a node that failed its checks (Part 2) or a known lemon.
Cluster packing simulator
Illustrative: a 64-GPU cluster and a stream of jobs of different sizes and priorities. Compare policies.
4. Slurm vs Kubernetes
Two families of software dominate. Slurm, from the HPC world, has long offered gang allocation (request eight GPUs across two nodes and you get them together or not at all), partitions, QoS, preemption, per-account limits and job accounting. Kubernetes, from cloud services, offers containers, rich automation and a huge ecosystem, but its default scheduler places pods one at a time: it has no gang scheduling, which can deadlock partially allocated jobs and fragment GPUs, and it does not know about NVLink or NUMA topology. Add-ons close much of the gap: Kueue (a Kubernetes SIG project) adds job queueing, Volcano adds gang and batch scheduling, and Dynamic Resource Allocation gives the scheduler structured knowledge of devices.
Increasingly, operators run both. SchedMD's Slinky runs Slurm on Kubernetes, CoreWeave offers SUNK and Nebius offers Soperator. In December 2025 NVIDIA acquired SchedMD, the company behind Slurm, and said Slurm would remain open-source and vendor-neutral; it had earlier bought and open-sourced the Kubernetes scheduler Run:ai โ ๏ธ secondary.
Which stack fits?
Simplified comparison from project docs and comparisons (some written by vendors).
5. Case study: Meta's MAST
Meta's problem was bigger than one cluster. Its GPUs sit in data centres across several regions, and when each team chose where to run, some regions were overloaded while others had spare capacity. Meta's answer, described in its OSDI 2024 paper, is MAST (a global scheduler for ML training across geo-distributed data centres). Users simply submit training jobs, and MAST decides where both the data and the training should go. It rests on three design principles: temporal decoupling (separating decisions made on different time scales), scope decoupling (separating global from local decisions) and exhaustive search over placement options. The headline result: in the most overloaded region, the ratio of high-priority GPU demand to supply fell from 2.63 to 0.98, effectively eliminating the overload.
Balancing regions
Illustrative regions. The busiest region's 2.63 โ 0.98 matches the paper; the others are invented to show the idea.
6. Gang scheduling and fragmentation
The most frustrating sight in an AI factory: a 512-GPU job waiting in the queue while 600 GPUs sit idle, scattered in small holes across many racks. Because the job needs its GPUs together, and preferably close together, scattered capacity is useless to it. Schedulers fight fragmentation with placement rules (pack small jobs together), backfilling (run short jobs in holes that will close in time) and occasional defragmentation (draining and repacking).
Fragmentation demo
Illustrative: 8 racks ร 16 GPUs. A 64-GPU job needs four whole free racks' worth of contiguous GPUs.
7. When it breaks
Scheduling detective
Pick a symptom. Simplified from operator practice.
8. Handoffs and careers
Who the platform team works with
- Typical backgrounds
- HPC administrators who know Slurm, Kubernetes and platform engineers from cloud companies, distributed-systems engineers.
- Core skills
- Slurm configuration and plugins, Kubernetes operators and schedulers (Kueue, Volcano), queueing and fairness policy, topology modelling, Go/Python, capacity planning, explaining trade-offs to research leads.
- Growth path
- Platform engineer โ owner of scheduling policy for a cluster โ architect of a global capacity system like MAST.
- Further reading
- Compute Economics ยท Part 4: Storage ยท Keeping the Factory Running
๐๏ธ Case file: Meta
- Scheduler
- MAST: global scheduling of ML training across geo-distributed data centres (OSDI 2024).
- Principles
- Temporal decoupling, scope decoupling, exhaustive search.
- Result
- Most overloaded region: high-priority demand/supply 2.63 โ 0.98.
- Llama 3 context
- 16K-GPU jobs on a 24K-GPU cluster with 3,072-GPU pods; placement within pods matters.
๐ Next in the series
Part 6: Kernels and compilers โ the GPU kernel and compiler engineers who write CUDA and Triton, tune FlashAttention and make every FLOP count.
๐ข Reading the numbers
- MAST facts come from the OSDI 2024 paper's abstract and USENIX page.
- NVIDIAโSchedMD comes from coverage by The Register and others (December 2025); Run:ai is โ ๏ธ secondary.
- Slurm/Kubernetes capabilities come from project docs and comparisons by SkyPilot, vCluster, Nebius and others, which have commercial interests.
- Role descriptions come from Perplexity, Crusoe and other postings.
- The packing simulator, region map (except the 2.63 โ 0.98 region) and fragmentation demo are illustrations.
๐ References
- Meta
- Choudhury et al.: "MAST: Global Scheduling of ML Training across Geo-Distributed Datacenters at Hyperscale" (OSDI 2024)
- Schedulers
- The Register: NVIDIA acquires SchedMD, pledges openness for Slurm
- SchedMD: Slinky, Slurm on Kubernetes (KubeCon Europe 2025)
- Nebius: Slurm vs Kubernetes for model training
- SkyPilot: Slurm vs Kubernetes ยท vCluster: Kubernetes schedulers vs Slurm
- Roles
- Perplexity: Member of Technical Staff, GPU Cluster Infrastructure (posting)
- Crusoe: Staff Software Engineer, Slurm (posting)
- Related on this site
- Part 4: Storage ยท Compute Economics ยท Producing Intelligence Part 1