Home โ€บ Blog โ€บ Scheduling and Orchestration
Who Runs the AI Factory ยท Part 5 ยท Scheduling

Who Gets the GPUs? The Engineers Who Schedule an AI Factory

An AI factory never has enough GPUs. A frontier pre-training run wants thousands of them for months, researchers want a few hundred for an afternoon, inference wants guaranteed capacity, and evaluation wants everything right before a launch. Someone has to decide who runs where and when, and do it so that expensive hardware is never idle. This part follows the cluster platform engineers who build and run the schedulers, from Slurm and Kubernetes to Meta's global scheduler, MAST.

FL
FrontierAGI Team
โš ๏ธ How this was researched Sources are Meta's OSDI 2024 paper on MAST, coverage of NVIDIA's acquisition of SchedMD, Slurm and Kubernetes project documentation and comparisons (several written by vendors with commercial interests, which we note), and real job postings (no OpenAI, Anthropic or xAI scheduling postings were found). The Run:ai detail is โš ๏ธ secondary. The packing simulator, region map and fragmentation demo are illustrations. See Reading the numbers.

1. Where we are in the stack

The AI factory stack, by role

2. The role: cluster platform engineer

Postings for this role blend HPC and cloud-native skills. Perplexity's Member of Technical Staff, GPU Cluster Infrastructure asks for deep Kubernetes expertise, including custom operators and multi-cluster management, to build a self-serve compute platform for inference engineers and researchers. A staff HPC posting centres on Slurm cluster architecture and multi-tenant scheduling policies, including modelling the physical network fabric for GPU scheduling and running Slurm on Kubernetes with Slinky. Crusoe's Staff Software Engineer, Slurm lists partitions, QoS, preemption, accounting and container integration. Typical asks are 5โ€“8+ years in infrastructure with substantial GPU, HPC or ML cluster experience, plus familiarity with NCCL, RDMA networking and GPU driver stacks.

What the role needs

Our reading of the postings (1โ€“5). The day timeline is a composite illustration.

3. The scheduling problem

A GPU scheduler has to satisfy requirements that pull against each other:

Cluster packing simulator

Illustrative: a 64-GPU cluster and a stream of jobs of different sizes and priorities. Compare policies.

4. Slurm vs Kubernetes

Two families of software dominate. Slurm, from the HPC world, has long offered gang allocation (request eight GPUs across two nodes and you get them together or not at all), partitions, QoS, preemption, per-account limits and job accounting. Kubernetes, from cloud services, offers containers, rich automation and a huge ecosystem, but its default scheduler places pods one at a time: it has no gang scheduling, which can deadlock partially allocated jobs and fragment GPUs, and it does not know about NVLink or NUMA topology. Add-ons close much of the gap: Kueue (a Kubernetes SIG project) adds job queueing, Volcano adds gang and batch scheduling, and Dynamic Resource Allocation gives the scheduler structured knowledge of devices.

Increasingly, operators run both. SchedMD's Slinky runs Slurm on Kubernetes, CoreWeave offers SUNK and Nebius offers Soperator. In December 2025 NVIDIA acquired SchedMD, the company behind Slurm, and said Slurm would remain open-source and vendor-neutral; it had earlier bought and open-sourced the Kubernetes scheduler Run:ai โš ๏ธ secondary.

Which stack fits?

Simplified comparison from project docs and comparisons (some written by vendors).

5. Case study: Meta's MAST

Meta's problem was bigger than one cluster. Its GPUs sit in data centres across several regions, and when each team chose where to run, some regions were overloaded while others had spare capacity. Meta's answer, described in its OSDI 2024 paper, is MAST (a global scheduler for ML training across geo-distributed data centres). Users simply submit training jobs, and MAST decides where both the data and the training should go. It rests on three design principles: temporal decoupling (separating decisions made on different time scales), scope decoupling (separating global from local decisions) and exhaustive search over placement options. The headline result: in the most overloaded region, the ratio of high-priority GPU demand to supply fell from 2.63 to 0.98, effectively eliminating the overload.

Balancing regions

Illustrative regions. The busiest region's 2.63 โ†’ 0.98 matches the paper; the others are invented to show the idea.

6. Gang scheduling and fragmentation

The most frustrating sight in an AI factory: a 512-GPU job waiting in the queue while 600 GPUs sit idle, scattered in small holes across many racks. Because the job needs its GPUs together, and preferably close together, scattered capacity is useless to it. Schedulers fight fragmentation with placement rules (pack small jobs together), backfilling (run short jobs in holes that will close in time) and occasional defragmentation (draining and repacking).

Fragmentation demo

Illustrative: 8 racks ร— 16 GPUs. A 64-GPU job needs four whole free racks' worth of contiguous GPUs.

7. When it breaks

Scheduling detective

Pick a symptom. Simplified from operator practice.

8. Handoffs and careers

Who the platform team works with

๐Ÿงญ Getting into the role
Typical backgrounds
HPC administrators who know Slurm, Kubernetes and platform engineers from cloud companies, distributed-systems engineers.
Core skills
Slurm configuration and plugins, Kubernetes operators and schedulers (Kueue, Volcano), queueing and fairness policy, topology modelling, Go/Python, capacity planning, explaining trade-offs to research leads.
Growth path
Platform engineer โ†’ owner of scheduling policy for a cluster โ†’ architect of a global capacity system like MAST.
Further reading
Compute Economics ยท Part 4: Storage ยท Keeping the Factory Running

๐Ÿ—‚๏ธ Case file: Meta

Scheduler
MAST: global scheduling of ML training across geo-distributed data centres (OSDI 2024).
Principles
Temporal decoupling, scope decoupling, exhaustive search.
Result
Most overloaded region: high-priority demand/supply 2.63 โ†’ 0.98.
Llama 3 context
16K-GPU jobs on a 24K-GPU cluster with 3,072-GPU pods; placement within pods matters.
Verdict Scheduling is where an AI factory's economics are won or lost: every idle GPU-hour is money, and every badly placed job runs slower. The field is converging on hybrids that keep Slurm's batch semantics on Kubernetes' automation, while the largest operators add a global layer, like Meta's MAST, that treats several data centres as one pool. The job is part systems engineering, part policy and part diplomacy.

๐Ÿ‘€ Next in the series

Part 6: Kernels and compilers โ€” the GPU kernel and compiler engineers who write CUDA and Triton, tune FlashAttention and make every FLOP count.

๐Ÿ”ข Reading the numbers

๐Ÿ“š References