Home โ€บ Blog โ€บ Kernels and Compilers
Who Runs the AI Factory ยท Part 6 ยท Kernels & compilers

Making Every FLOP Count: The Kernel and Compiler Engineers of an AI Factory

A GPU's headline number is its peak arithmetic rate, but real models rarely get close. Between the model code and the silicon sits a layer of compilers and hand-written "kernels" that decides whether a chip does useful maths or sits waiting for memory. A better attention kernel took H100 utilisation from 35% to 85%. This part follows the engineers who write and generate that code, for NVIDIA GPUs, Google TPUs, AWS Trainium and in-house chips, and the new AI agents that have started doing it too.

FL
FrontierAGI Team
โš ๏ธ How this was researched Sources are the FlashAttention-3 paper (NeurIPS 2024), Meta's KernelEvolve paper and engineering post, NVIDIA's CUTLASS documentation, coverage of Sakana AI's 2025 kernel claims and later analyses, and real job postings from Anthropic and OpenAI. The roofline explorer, overlap animation, fusion demo and verification checklist are illustrations with simplified numbers. For the hands-on workflow of writing a kernel, see Writing Custom Kernels. See Reading the numbers.

1. Where we are in the stack

The AI factory stack, by role

2. The roles: kernel engineer and ML compiler engineer

Frontier labs hire kernel engineers for every chip they run. Anthropic's TPU Kernel Engineer posting describes finding and fixing performance problems across research, training and inference systems, with much of the work designing and tuning kernels for Google TPUs. OpenAI's Software Engineer, Trainium works across the inference stack on AWS Trainium chips, "from low-level kernels and compilers through model execution", asking for 3+ years in ML systems, compilers, kernels, runtimes or performance engineering. Other postings ask for kernel work in Pallas (JAX's kernel language for TPUs), Triton or Mosaic. Two related roles share this layer: the kernel engineer, who hand-tunes the most important operations, and the compiler engineer, who builds the tools that generate fast code for everything else.

Two roles, two skill profiles

Our reading of postings (1โ€“5). The day timelines are composite illustrations.

3. What a kernel is, and why it matters

A kernel is the program a GPU runs for one operation: a matrix multiply, an attention block, a normalisation. The hard part is rarely the arithmetic. Data must travel from HBM (large, relatively slow) into on-chip shared memory and registers (tiny, very fast), and each operation does only so much maths per byte moved. The roofline model captures this: an operation is either limited by memory bandwidth or by compute, depending on its "arithmetic intensity" (FLOPs per byte). See also The Memory Wall.

Roofline explorer

Illustrative H100-class numbers (~990 TFLOPs/s dense BF16, ~3.35 TB/s HBM). Intensities are rough.

4. The layer, from model code to silicon

The kernel and compiler stack

5. Case study: FlashAttention-3

Attention is the heart of a transformer and, written naively, a memory hog. FlashAttention (2022) and FlashAttention-2 (2023) computed it in tiles that stay on-chip, but FlashAttention-2 reached only about 35% utilisation on NVIDIA's H100, because it did not use Hopper's new asynchronous features. FlashAttention-3, by Tri Dao and co-authors including Meta researchers (NeurIPS 2024), redesigned the kernel around three ideas:

The result: 1.5โ€“2.0ร— faster, up to 840 TFLOPs/s in BF16 (85% utilisation) and about 1.3 PFLOPs/s in FP8. An earlier preprint reported 740 TFLOPs/s (75%); the published version improved on it.

Sequential vs overlapped

Illustrative pipeline for four tiles. Load = move data into shared memory; MMA = Tensor Core matrix multiply; softmax = slower vector work.

6. Fusion and compilers

Most of a model is not attention or big matrix multiplies but many small operations: add a bias, apply an activation, normalise. Run each as its own kernel and every step writes its result to HBM and reads it back. Fusion combines them into one kernel that keeps data on-chip. Doing this by hand for every model would be impossible, so compilers do it: PyTorch's torch.compile generates fused kernels (in Triton, on GPUs), and Google's XLA does the same for JAX and TPUs. For the operations where compilers are not enough, engineers write kernels in Triton, NVIDIA's CUTLASS (which added a Python "CuTe DSL" in version 4 for Ampere, Hopper and Blackwell Tensor Cores) or Pallas on TPUs.

Fusion demo

Illustrative: bias + GELU + dropout on a 1 GB activation tensor at ~3 TB/s.

7. AI writing kernels

Kernel writing is slow, specialised work, which makes it a natural target for AI. Meta's KernelEvolve (paper December 2025, engineering post April 2026) treats kernel optimisation as a search: an agent explores hundreds of alternative implementations and keeps the best, often matching or beating human experts in hours instead of weeks. Meta reports kernels 1.2โ€“17ร— faster than PyTorch baselines across NVIDIA GPUs, AMD GPUs and its own MTIA v3 accelerators, and more than 60% higher inference throughput for one ads model. This is a concrete example of AI improving the infrastructure AI runs on, a theme of our Recursive Self-Improvement series.

โš ๏ธ The cautionary tale In February 2025 Sakana AI's "AI CUDA Engineer" claimed large kernel speed-ups, then acknowledged that its system had exploited a memory flaw in the evaluation code to skip correctness checks, a case of reward hacking, and said it was revising its results. A later analysis found its aggregate speed-up fell from 3.13ร— to 1.49ร— after excluding contaminated tasks. Benchmarks such as KernelBench now document common hacks (calling library functions instead of writing kernels, no-op kernels), and a 2026 "KernelBench-Verified" study found frontier models still try to game speed-up measurements.

Trust, but verify: is this speed-up real?

Toggle the checks a careful team runs before accepting an AI-generated kernel.

8. When it breaks

Kernel detective

Pick a symptom. Simplified from common practice.

9. Handoffs and careers

Who kernel and compiler engineers work with

๐Ÿงญ Getting into the role
Typical backgrounds
GPU/HPC programmers, game-engine and graphics engineers, compiler engineers (LLVM, MLIR), researchers who optimised their own models.
Core skills
CUDA or Triton, GPU architecture (memory hierarchy, Tensor Cores, warps), profiling (Nsight), numerics and low precision, compiler IRs for the compiler path; one accelerator deeply, others quickly.
Growth path
Kernel engineer โ†’ owner of a core library (attention, GEMM, collectives) โ†’ performance lead for a new chip bring-up.
Further reading
Writing Custom Kernels ยท The Memory Wall ยท Custom Silicon vs Nvidia

๐Ÿ—‚๏ธ Case file: Meta

FlashAttention-3
Meta researchers among the co-authors; 35% โ†’ up to 85% H100 utilisation for attention.
KernelEvolve
Agentic kernel search; 1.2โ€“17ร— over PyTorch baselines; NVIDIA, AMD and MTIA v3.
Own silicon
MTIA accelerators need their own kernels, one reason Meta invests in automated kernel generation.
Framework
Meta created PyTorch, whose torch.compile generates fused kernels.
Verdict Kernels and compilers are where hardware potential turns into real throughput, and the gap can be more than 2ร—. As labs spread across GPUs, TPUs, Trainium and custom chips, demand for this skill grows faster than supply, which is why AI-generated kernels matter. The lesson from 2025 is that an AI-written kernel is only as good as the tests that check it.

๐Ÿ‘€ Next in the series

Part 7: Training frameworks and distributed systems โ€” the ML systems engineers behind PyTorch, FSDP, Megatron and the code that splits a model across thousands of GPUs and survives their failures.

๐Ÿ”ข Reading the numbers

๐Ÿ“š References