Writing Custom Kernels and Why You Occasionally Have To
Article 4 introduced Model FLOPs Utilization as the honest measure of whether a GPU is doing useful work. This article covers what happens after you find a job with stubbornly low MFU that framework-level fixes can't touch: why PyTorch's built-in operators sometimes aren't fast enough, what a GPU kernel actually is, why Triton has become the practical entry point for most engineers instead of raw CUDA, and the profiling discipline that separates a justified custom kernel from wasted engineering effort.
The Layer Almost Nobody Touches, Until Someone Has To
Most research engineering work in this series so far happens entirely in Python: writing training loops, wiring up configs, debugging distributed jobs, and reasoning about scheduler queues. All of that sits on top of an assumption — that when your code calls a matrix multiply or an attention operation, the underlying GPU implementation is already about as fast as it can be. Most of the time that assumption holds, because PyTorch's built-in operators are themselves backed by heavily-optimized vendor kernels. Occasionally it doesn't, and understanding why is what separates an engineer who can only tune hyperparameters from one who can actually make a slow model fast. This article is deliberately narrow: it is not a CUDA programming course, and it will not make you a kernel engineer. It is the honest map of when this level of the stack matters, what the entry point actually looks like today, and — just as importantly — when reaching for it is the wrong move.
The Question That Comes Before Any Optimization
Horace He's widely-cited engineering post, "Making Deep Learning Go Brrrr From First Principles", frames the entire problem cleanly: every GPU operation is either compute-bound (the GPU's arithmetic units are the bottleneck — it's actually crunching numbers as fast as it can) or memory-bound (the GPU is mostly waiting on data moving between its slow high-bandwidth memory and its fast on-chip memory, with the arithmetic units sitting comparatively idle). This is the same "is the GPU on vs. is the GPU doing useful work" distinction from Article 4's MFU discussion, pushed one level deeper: a memory-bound operation can show high GPU utilization in a naive sense while still wasting most of its theoretical compute capacity, because the numbers it's waiting to receive are the actual constraint, not the arithmetic itself.
This matters because it inverts a common intuition. Many engineers assume the path to a faster model is always "fewer floating-point operations" — a smaller, more elegant formula. But if an operation is memory-bound, an algebraically simpler formula that still moves the same amount of data through memory won't help at all. This is precisely why operations like attention — mathematically simple, but involving large intermediate tensors that don't fit in fast on-chip memory — became one of the first and most famous custom-kernel success stories, covered in Part 4 below.
The Discipline That Prevents Wasted Weeks
The single most common mistake at this level of the stack is writing a custom kernel for an operation that was never actually the bottleneck. PyTorch's own torch.profiler documentation exists precisely because "which operation is slow" is an empirical question, not something you can reliably guess from reading code — a training step involves dozens of operations, and intuition about which one dominates wall-clock time is wrong often enough that skipping this step routinely wastes engineering time on the wrong target.
The pattern that recurs across real engineering accounts is: profile first, identify the specific operation and its bottleneck type, estimate the realistic ceiling on improvement (a memory-bound op is bounded by memory bandwidth, not by however clever the replacement code is), and only then decide whether a custom kernel is worth the engineering and maintenance cost — a cost this article returns to directly in Part 5.
Why Most Engineers Never Touch Raw CUDA
OpenAI's "Introducing Triton: Open-Source GPU Programming for Neural Networks" describes the actual reason Triton has become the practical entry point for this work: writing efficient raw CUDA requires reasoning explicitly about memory hierarchies, thread scheduling, and hardware-specific tuning that historically took specialist GPU engineers years to master. Triton lets an engineer write Python-like code that the compiler lowers to efficient GPU code automatically handling much of that low-level complexity, trading some of the theoretical peak performance a hand-tuned CUDA expert could reach for a dramatically shorter path to a working, meaningfully-faster kernel.
| Approach | Learning Curve | Typical Use Case |
|---|---|---|
| PyTorch built-in ops | None — this is the default | The vast majority of code; vendor-optimized already |
| Triton | Moderate — Python-like syntax, GPU concepts still required | A specific profiled bottleneck where a fused custom op would help |
| Raw CUDA / C++ | Steep — genuine specialist skill | Extracting the last percentage points of peak hardware performance, novel hardware features |
It's also worth naming that a large share of what used to require hand-written kernels is now handled automatically by torch.compile, which can fuse operations and generate Triton kernels behind the scenes without an engineer writing GPU code directly at all — meaning the honest first question, before writing anything by hand, is whether the compiler can already close the gap.
A Real Kernel That Changed the Field
Dao et al.'s "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness" (2022) is the canonical real-world example of everything in Parts 1 through 3 combined: attention was identified as memory-bound (Part 1) rather than compute-bound, because standard implementations materialize large intermediate attention matrices that don't fit in fast GPU memory; the fix was not a new attention algorithm producing different numbers, but a kernel-level rewrite that restructures the computation to keep more of it in fast on-chip memory and avoid ever materializing the full intermediate matrix — the same mathematical result, computed with far less memory traffic. The paper reports substantial wall-clock speedups and memory reductions on real training and inference workloads, and its core technique was significant enough to be adopted directly into mainstream frameworks rather than remaining a standalone library.
The Cost Side of the Ledger
A custom kernel is not a free win. It is code that must be maintained across new GPU hardware generations, verified for numerical correctness against the reference implementation (an easy place for silent bugs, echoing Article 3's warning about training that runs without crashing while producing wrong numbers), and re-tuned as workload shapes change. Sculley et al.'s "Hidden Technical Debt in Machine Learning Systems" — already cited in Article 2 of this series for configuration sprawl — applies just as directly here: a hand-written kernel is exactly the kind of specialized, hard-to-read, high-maintenance code the paper warns accumulates debt fastest, and it should only be taken on when the profiled, quantified speedup clearly justifies that ongoing cost.
torch.compile or an existing library already solves it before writing anything by hand.Real Scenario Walkthroughs
Readiness Checklist
⚠️ What's Missing or Uncertain
torch.compile, profiler APIs) also change rapidly as frameworks evolve.
Where This Series Goes Next
Article 6 moves from the deepest, most specialized layer of the stack to the opposite end of the daily experience: the on-call reality of monitoring, alerts, and 2am incidents — what happens when a job like the ones profiled and optimized in this article fails in production, at an hour when the person paged is not the person who wrote the kernel.
- Horace He — "Making Deep Learning Go Brrrr From First Principles"
- PyTorch — torch.profiler documentation
- OpenAI — "Introducing Triton: Open-Source GPU Programming for Neural Networks"
- Dao et al. — "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness" (arXiv:2205.14135)
- Sculley et al. — "Hidden Technical Debt in Machine Learning Systems" (NeurIPS 2015)
- This site — Frontier Lab Engineering: Compute Economics