Frontier Lab Engineering Practicum · Article 4 of 9💳
Compute Economics: Requesting, Scheduling, and Not Wasting a GPU-Hour
GPU time is the scarcest, most expensive resource in this entire field, and almost nobody explains how it's actually allocated day to day. This article covers job scheduler queues, spot vs. reserved capacity, what "utilization" really measures, and why the person approving your compute request is applying real, quantifiable logic — not gatekeeping for its own sake.
Every article so far in this series has assumed you can get a GPU when you need one. In practice, that assumption is the single biggest constraint on a research engineer's daily productivity — more than code quality, more than model design. GPU clusters are shared, finite, expensive resources, and how they're allocated among many competing requests is itself a real, decades-old systems problem. Google's Borg paper (Verma et al., 2015) documents exactly this challenge at Google's scale — priorities, quotas, and preemption, the same three concepts that govern whether your training job starts this afternoon or sits in a queue until tomorrow.
3Core scheduling concepts every engineer eventually needs: priority, quota, preemption
MFUModel FLOPs Utilization — the actual metric introduced in Google's PaLM paper for "how efficiently is this job using its GPUs"
~0.17–3Real per-GPU-hour price range ($) across providers cited in Article 1 of the Foundations series
100%Idle GPU utilization that costs exactly the same as a fully utilized one — the core economic fact this article is about
Part 1 — How Compute Is Actually Requested
Priority, Quota, and Preemption
A shared GPU cluster (Article 10 of the Foundations series' scheduler layer) has to make a decision every time more jobs want GPUs than are physically available. Three mechanisms, all present in Google's Borg system and its descendants (including the open-source Slurm workload manager used widely in ML research), govern that decision:
1
Priority — not all jobs are equal; a time-sensitive production training run typically outranks an exploratory ablation, and the scheduler enforces this ordering automatically rather than leaving it to informal negotiation.
2
Quota — teams or individuals are usually allocated a maximum share of the cluster's total capacity, preventing one team's burst of activity from starving every other team indefinitely.
3
Preemption — a lower-priority running job can be paused (or killed) to free resources for a higher-priority one — which is exactly why Article 2 of this series' checkpointing discipline isn't optional politeness, it's the only thing that makes preemption survivable rather than catastrophic.
The scheduler's priority ladder — not arbitrary, but a real allocation mechanism a Borg-style cluster runs continuously.
Part 2 — Spot vs. Reserved Capacity
The Same Tradeoff, Every Cloud Provider
Article 1 of the Foundations series cited real 2026 pricing across Lambda Labs, RunPod, and Vast.ai — the same reserved-vs-spot tradeoff governs internal cluster allocation too, just with organizational priority standing in for a market price.
Capacity Type
Cost
Reliability
Best For
Reserved / on-demand
Highest
Guaranteed, won't be preempted
Production runs, anything with a hard deadline
Spot / preemptible
Lowest (often a fraction of on-demand)
Can be reclaimed with little or no warning
Exploratory work, anything checkpointed frequently enough to resume cheaply
The engineering skill this tradeoff actually demands isn't picking one option — it's matching the option to the job. A hyperparameter sweep across 50 small, frequently-checkpointed runs is a natural fit for spot capacity; a final release-candidate training run generally isn't, since a preemption mid-run on a job that matters is expensive in a way the spot discount rarely offsets.
Part 3 — What "Utilization" Actually Means
Not All Busy-Looking GPUs Are Actually Working Hard
A GPU can show high activity in a basic monitoring tool while still being deeply inefficient — waiting on data loading, waiting on communication (Article 10's ring-allreduce), or running with a batch size too small to use its full compute capacity. Chowdhery et al.'s PaLM paper (2022) introduced Model FLOPs Utilization (MFU) specifically to measure this more honestly: the ratio of floating-point operations actually being usefully performed for training versus the theoretical maximum the hardware could deliver. A training run reporting 45-55% MFU (the range PaLM itself reported and treated as a strong result) is doing meaningfully better than one at 15%, even if both look "busy" on a simple GPU-activity graph.
When someone asks about your job's "utilization," they're rarely asking whether the GPU is turned on — they're asking whether the expensive hardware is doing useful floating-point work or mostly waiting on something else.
Right-size your request. Requesting 64 GPUs for a job that can only actually use 8 efficiently (per Article 10's parallelism math) doesn't just waste your own budget — it holds capacity other teams' jobs are waiting on.
2
Release GPUs you're not using. An interactive debug session left open overnight, holding a GPU idle, is a real, visible cost on shared infrastructure in a way it wasn't on a personal laptop.
3
Checkpoint proactively, not reactively. Given Part 1's preemption mechanics, a job that only checkpoints at the very end effectively bets its entire runtime against ever being preempted — a bet that gets worse as job length and cluster contention both increase.
Part 5 — The Real Cost of a Wasted GPU-Hour
It's Not Just Money
This site's own Power Bottleneck investigation covers the infrastructure-economics side of this question in depth; the piece most directly relevant here is Patterson et al.'s "Carbon Emissions and Large Neural Network Training" (2021), which quantifies the real energy and carbon cost per unit of training compute — meaning a wasted GPU-hour (an idle debug session, an oversized job request, an unnecessarily preempted run that has to restart from scratch) has a measurable environmental cost on top of its measurable dollar cost, not just an abstract inefficiency.
Part 6 — Real Scenarios
Real Scenario Walkthroughs
📊Scenario A — A Job That "Looks Busy" But Is Actually Starved
A common realistic diagnosis: a training job shows high GPU activity in a basic dashboard, but its MFU (Part 3) is unexpectedly low. Following Article 3 of this series' debugging instincts, the likely cause is the GPU spending most cycles waiting — on a slow data loader (Article 1 of this series' data pipeline theme), or on communication overhead from a parallelism strategy mismatched to the job's actual size (Article 10 of the Foundations series). "The GPU is on" and "the GPU is doing useful work" are different claims, and only MFU distinguishes them.
The lesson: always check MFU, not just raw activity, before concluding a slow job is compute-bound rather than starved by something upstream.
⏳Scenario B — Choosing Spot Capacity for the Wrong Job
A team runs its single, final release-candidate training job on spot/preemptible capacity to save cost, and it gets preempted twice mid-run with limited checkpointing, costing more in lost progress and delay than the spot discount saved. Applying Part 2's matching principle after the fact makes the mistake legible: this was exactly the kind of high-stakes, low-tolerance-for-restart job that belonged on reserved capacity, regardless of the per-hour price difference.
The lesson: the cheaper option isn't automatically the better one — matching capacity type to the job's actual tolerance for interruption is the real skill, not simply minimizing the hourly rate.
Readiness Checklist
1
Can you explain the difference between priority, quota, and preemption in a shared cluster scheduler?
2
Can you explain what Model FLOPs Utilization measures, and why a "busy-looking" GPU can still have low MFU?
3
Given a specific job, can you reason about whether it belongs on spot or reserved capacity, rather than defaulting to whichever is cheaper?
4
Have you ever right-sized a compute request downward after realizing a job couldn't actually use all the GPUs requested?
⚠️ What's Missing or Uncertain
Specific scheduler policies, quota systems, and pricing vary significantly between organizations and cloud providers. The concepts in this article (priority, quota, preemption, MFU) are general and durable; the exact numbers and interfaces you'll encounter will differ from any single example here — treat this as the vocabulary and mental model, not a specific tool's manual.
Where This Series Goes Next
Article 5 moves from compute economics to the lowest level of the stack: writing custom kernels and understanding when Python-level code genuinely isn't fast enough, building on this article's utilization concepts (a low-MFU job is often exactly the signal that motivates a custom kernel in the first place) with real CUDA/Triton basics and profiling-before-optimizing discipline.
🧭 Closing — GPUs Are Money, Time, and Carbon at Once
🎯 The Bottom Line
A shared GPU cluster's priority, quota, and preemption mechanics — the same ones documented in Google's Borg system a decade ago — aren't bureaucracy standing between you and your job; they're the actual, necessary allocation logic for a resource every team wants more of than exists. Understanding Model FLOPs Utilization, matching spot versus reserved capacity to a job's real tolerance for interruption, and treating an idle GPU as a real cost — in dollars and in Patterson et al.'s measured carbon terms — is what separates an engineer who merely runs jobs from one whose manager trusts them with a growing compute budget.