Making Every FLOP Count: The Kernel and Compiler Engineers of an AI Factory
A GPU's headline number is its peak arithmetic rate, but real models rarely get close. Between the model code and the silicon sits a layer of compilers and hand-written "kernels" that decides whether a chip does useful maths or sits waiting for memory. A better attention kernel took H100 utilisation from 35% to 85%. This part follows the engineers who write and generate that code, for NVIDIA GPUs, Google TPUs, AWS Trainium and in-house chips, and the new AI agents that have started doing it too.
1. Where we are in the stack
The AI factory stack, by role
2. The roles: kernel engineer and ML compiler engineer
Frontier labs hire kernel engineers for every chip they run. Anthropic's TPU Kernel Engineer posting describes finding and fixing performance problems across research, training and inference systems, with much of the work designing and tuning kernels for Google TPUs. OpenAI's Software Engineer, Trainium works across the inference stack on AWS Trainium chips, "from low-level kernels and compilers through model execution", asking for 3+ years in ML systems, compilers, kernels, runtimes or performance engineering. Other postings ask for kernel work in Pallas (JAX's kernel language for TPUs), Triton or Mosaic. Two related roles share this layer: the kernel engineer, who hand-tunes the most important operations, and the compiler engineer, who builds the tools that generate fast code for everything else.
Two roles, two skill profiles
Our reading of postings (1โ5). The day timelines are composite illustrations.
3. What a kernel is, and why it matters
A kernel is the program a GPU runs for one operation: a matrix multiply, an attention block, a normalisation. The hard part is rarely the arithmetic. Data must travel from HBM (large, relatively slow) into on-chip shared memory and registers (tiny, very fast), and each operation does only so much maths per byte moved. The roofline model captures this: an operation is either limited by memory bandwidth or by compute, depending on its "arithmetic intensity" (FLOPs per byte). See also The Memory Wall.
Roofline explorer
Illustrative H100-class numbers (~990 TFLOPs/s dense BF16, ~3.35 TB/s HBM). Intensities are rough.
4. The layer, from model code to silicon
The kernel and compiler stack
5. Case study: FlashAttention-3
Attention is the heart of a transformer and, written naively, a memory hog. FlashAttention (2022) and FlashAttention-2 (2023) computed it in tiles that stay on-chip, but FlashAttention-2 reached only about 35% utilisation on NVIDIA's H100, because it did not use Hopper's new asynchronous features. FlashAttention-3, by Tri Dao and co-authors including Meta researchers (NeurIPS 2024), redesigned the kernel around three ideas:
- Asynchrony and warp specialisation: some groups of threads (warps) only load data using the Tensor Memory Accelerator while others only compute, so loading and maths overlap.
- Interleaving matrix multiplies and softmax: the slower softmax runs while the Tensor Cores work on the next block.
- Low precision done carefully: FP8 with techniques that cut numerical error 2.6ร compared with a baseline FP8 attention.
The result: 1.5โ2.0ร faster, up to 840 TFLOPs/s in BF16 (85% utilisation) and about 1.3 PFLOPs/s in FP8. An earlier preprint reported 740 TFLOPs/s (75%); the published version improved on it.
Sequential vs overlapped
Illustrative pipeline for four tiles. Load = move data into shared memory; MMA = Tensor Core matrix multiply; softmax = slower vector work.
6. Fusion and compilers
Most of a model is not attention or big matrix multiplies but many small operations: add a bias, apply an activation, normalise. Run each as its own kernel and every step writes its result to HBM and reads it back. Fusion combines them into one kernel that keeps data on-chip. Doing this by hand for every model would be impossible, so compilers do it: PyTorch's torch.compile generates fused kernels (in Triton, on GPUs), and Google's XLA does the same for JAX and TPUs. For the operations where compilers are not enough, engineers write kernels in Triton, NVIDIA's CUTLASS (which added a Python "CuTe DSL" in version 4 for Ampere, Hopper and Blackwell Tensor Cores) or Pallas on TPUs.
Fusion demo
Illustrative: bias + GELU + dropout on a 1 GB activation tensor at ~3 TB/s.
7. AI writing kernels
Kernel writing is slow, specialised work, which makes it a natural target for AI. Meta's KernelEvolve (paper December 2025, engineering post April 2026) treats kernel optimisation as a search: an agent explores hundreds of alternative implementations and keeps the best, often matching or beating human experts in hours instead of weeks. Meta reports kernels 1.2โ17ร faster than PyTorch baselines across NVIDIA GPUs, AMD GPUs and its own MTIA v3 accelerators, and more than 60% higher inference throughput for one ads model. This is a concrete example of AI improving the infrastructure AI runs on, a theme of our Recursive Self-Improvement series.
Trust, but verify: is this speed-up real?
Toggle the checks a careful team runs before accepting an AI-generated kernel.
8. When it breaks
Kernel detective
Pick a symptom. Simplified from common practice.
9. Handoffs and careers
Who kernel and compiler engineers work with
- Typical backgrounds
- GPU/HPC programmers, game-engine and graphics engineers, compiler engineers (LLVM, MLIR), researchers who optimised their own models.
- Core skills
- CUDA or Triton, GPU architecture (memory hierarchy, Tensor Cores, warps), profiling (Nsight), numerics and low precision, compiler IRs for the compiler path; one accelerator deeply, others quickly.
- Growth path
- Kernel engineer โ owner of a core library (attention, GEMM, collectives) โ performance lead for a new chip bring-up.
- Further reading
- Writing Custom Kernels ยท The Memory Wall ยท Custom Silicon vs Nvidia
๐๏ธ Case file: Meta
- FlashAttention-3
- Meta researchers among the co-authors; 35% โ up to 85% H100 utilisation for attention.
- KernelEvolve
- Agentic kernel search; 1.2โ17ร over PyTorch baselines; NVIDIA, AMD and MTIA v3.
- Own silicon
- MTIA accelerators need their own kernels, one reason Meta invests in automated kernel generation.
- Framework
- Meta created PyTorch, whose torch.compile generates fused kernels.
๐ Next in the series
Part 7: Training frameworks and distributed systems โ the ML systems engineers behind PyTorch, FSDP, Megatron and the code that splits a model across thousands of GPUs and survives their failures.
๐ข Reading the numbers
- FlashAttention-3 figures come from the NeurIPS 2024 paper (and its preprint for the 740 TFLOPs/s figure).
- KernelEvolve figures come from Meta's paper and engineering post.
- CuTe DSL facts come from NVIDIA's CUTLASS documentation; its release status (beta vs general availability) was unclear in our sources.
- Sakana AI facts come from TechCrunch coverage, Sakana's statement and later analyses (3.13ร โ 1.49ร).
- Role descriptions come from Anthropic and OpenAI postings (which may be closed).
- Roofline, overlap, fusion and verification demos are illustrations with simplified numbers.
๐ References
- Kernels
- Shah et al.: "FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision" (NeurIPS 2024)
- NVIDIA CUTLASS overview (CuTe DSL)
- AI-generated kernels
- Meta: "KernelEvolve: Scaling Agentic Kernel Coding for Heterogeneous AI Accelerators" (2025)
- Meta Engineering: KernelEvolve (April 2026)
- TechCrunch: Sakana walks back claims about its AI CUDA Engineer
- EvoEngineer: automated CUDA kernel evolution (replication analysis)
- KernelBench-Verified: Do LLM-generated kernels actually beat PyTorch? (2026)
- Roles
- Anthropic: TPU Kernel Engineer (posting)
- OpenAI: Software Engineer, Trainium (posting)
- Related on this site
- Writing Custom Kernels ยท Part 5: Scheduling ยท The Memory Wall