The Kernels That Matter: FlashAttention, CUTLASS, Triton and the Code That Actually Runs on the GPU
Frameworks describe the maths and parallelism libraries split it across thousands of chips, but the arithmetic itself happens inside kernels: small programs launched across tens of thousands of GPU threads at once. A handful of them, mostly matrix multiplies and attention, account for most of an AI factory's compute. Whether they reach 35% or 75% of a chip's peak decides how many GPUs a lab needs. This part explains why, how the best kernels are written, and who writes them.
1. Case file
2. Timeline
GPU kernels for AI, 2007 โ 2026
Tap a milestone. Colour shows the family: NVIDIA libraries, attention, languages, open-source labs.
3. The GPU in one picture
A modern data-centre GPU is about a hundred independent processors (streaming multiprocessors, SMs; 132 on an H100 SXM) sharing a large, slow pool of high-bandwidth memory (HBM). Each SM has tensor cores for matrix maths, a small fast scratchpad (shared memory) and registers. Data gets faster and scarcer the closer it is to the arithmetic, and a kernel's job is to move each number up the ladder as few times as possible.
The memory ladder (H100 SXM)
Tap a level. Capacity and bandwidth are approximate; on-chip bandwidths are aggregate across all SMs and vary with access pattern.
4. Why kernels decide speed: the roofline
Every kernel is limited either by arithmetic or by memory traffic. The deciding number is arithmetic intensity: floating-point operations per byte moved from HBM. Below the chip's "ridge point" (peak FLOPS รท bandwidth, about 295 FLOPs per byte on an H100) a kernel is memory-bound and extra tensor cores do nothing; above it, the kernel is compute-bound. Big matrix multiplies sit far to the right. Softmax, layer norm and token-by-token decoding sit far to the left, which is why they are fused and why inference is so often a memory problem.
Interactive roofline
Pick a GPU and an operation. Logโlog axes. Intensities are rounded illustrations for typical LLM shapes in BF16.
5. Matrix multiply, the workhorse
Most of a transformer's FLOPs are matrix multiplies (GEMMs): the attention projections and the feed-forward layers. The trick that makes them fast is tiling: load a block of each input into shared memory once, and reuse it for many multiply-adds before fetching the next block. Larger tiles mean more reuse and fewer trips to HBM, up to the limit of on-chip memory.
Tiling animator
C = A ร B with 8 ร 8 matrices. Each step computes one output tile. Count how many input numbers are fetched from HBM. Our own illustration.
On NVIDIA hardware, three layers of tools produce GEMMs. cuBLAS and cuBLASLt are NVIDIA's closed, pre-tuned libraries that frameworks call by default. CUTLASS (2017) is NVIDIA's open-source C++ template library exposing the same building blocks, and CuTe is its layout algebra for describing how tiles map onto threads and memory. Hopper (H100) added hardware that the best kernels must use: the Tensor Memory Accelerator (TMA) copies tiles between HBM and shared memory asynchronously, warpgroup MMA (wgmma) instructions feed the tensor cores from shared memory, and warp specialisation dedicates some threads to loading data while others compute. Blackwell adds tensor memory and two-SM cooperative MMA. Each generation makes older kernels leave more performance on the table.
6. FlashAttention, step by step
Standard attention computes a score for every pair of tokens: an N ร N matrix, written to HBM, read back for the softmax, written again, read again to multiply by V. At long sequence lengths that matrix dominates both memory and time. FlashAttention (Tri Dao and colleagues, 2022) never materialises it. It loads blocks of queries, keys and values into shared memory, computes partial scores, and keeps a running maximum and sum so the softmax can be corrected as each new block arrives (the "online softmax"). The result is mathematically exact, uses memory proportional to N instead of Nยฒ, and reads HBM far less.
Standard vs Flash attention
Step through each algorithm, then move the sequence-length slider to see memory for the score matrix. One head, BF16. Illustration.
Four generations
Peak forward-pass figures reported by each paper on the GPU it targeted. Different chips; read as progress, not a like-for-like race.
FlashAttention-2 ~35% on H100 and FlashAttention-3 740 TFLOPS (75%) are from the FA3 paper; FA3 FP8 "close to 1.2 PFLOPS". FA4: up to 1,613 TFLOPS BF16 on B200 (~71%), 1.3ร cuDNN 9.13 and 2.7ร Triton, per the FA4 paper.
- FlashAttention (2022): tiling plus online softmax; exact attention with linear memory.
- FlashAttention-2 (2023): better work partitioning across threads and over the sequence; about twice as fast on A100.
- FlashAttention-3 (Jul 2024): rewritten for Hopper with TMA, wgmma, warp specialisation and overlapping softmax with matrix multiplies; FP8 support with 2.6ร lower numerical error than a baseline FP8 attention.
- FlashAttention-4 (Mar 2026): co-designed for Blackwell, where tensor-core throughput doubled but shared-memory bandwidth and exponential units did not; uses software-emulated exponentials, larger asynchronous tiles and tensor memory. Written entirely in CuTe-DSL, CUTLASS's Python kernel language, with 20โ30ร faster compile times than C++ templates (paper).
7. Fusion
Three kernels or one?
y = dropout(gelu(x + bias)) over a 4,096 ร 16,384 BF16 activation (64M elements, 128 MB). Count HBM traffic. Our arithmetic, ignoring caches.
Memory-bound operations gain almost nothing from faster arithmetic, but everything from not touching HBM. Fusing them into one kernel, or into the epilogue of the preceding matrix multiply, is the cheapest speedup in the stack. Compilers now do much of this automatically: TorchInductor (Part 2) generates fused Triton kernels and XLA fuses whole programs. Hand-written fused kernels remain common for the hottest paths: attention, normalisation plus quantisation, mixture-of-experts routing and the inference engines' paged attention (Part 7).
8. Ways to write kernels
The same fused row-softmax, three ways
Simplified; real kernels add masking, tiling across long rows and launch code.
| Tool | Level | Who | Best for |
|---|---|---|---|
| CUDA C++ | Lowest: threads, warps, PTX | NVIDIA | Maximum control; every NVIDIA feature first |
| CUTLASS / CuTe (C++) | Templates over CUDA | NVIDIA, open source | Peak GEMMs and attention; FA3, many vendor kernels |
| CuTe-DSL (Python) | Python front-end to CuTe | NVIDIA | The same control with fast compiles; FA4 |
| Triton | Python, block-level | OpenAI, open source | Fast to write, good performance; torch.compile backend; runs on NVIDIA and AMD |
| ThunderKittens | Small embedded DSL in CUDA, tile-level | Stanford Hazy Research | Simple, fast attention-style kernels (May 2024) |
| Pallas | Python, inside JAX | Custom kernels on TPUs (via Mosaic) and GPUs | |
| NKI โ ๏ธ | Python | AWS | Custom kernels on Trainium |
9. Low precision
Fewer bits, more FLOPS
Dense tensor-core peaks per GPU from NVIDIA datasheets, approximate. Lower precision roughly doubles throughput each step but needs careful scaling to keep accuracy.
Lower precision is now a kernel problem as much as a hardware one. FP8 has a tiny range, so values are stored with scaling factors; how finely those scales are applied decides accuracy. DeepSeek's DeepGEMM (open-sourced Feb 2025) uses fine-grained, block-wise FP8 scaling and promotes partial sums to higher precision on CUDA cores to limit error, reaching up to about 1,550 TFLOPS on an H800 according to its repository. Blackwell adds 4-bit formats (FP4, and NVIDIA's NVFP4 with small-block scaling โ ๏ธ), mostly used for inference so far. FlashAttention-3's FP8 path illustrates the trade-off: it reduced numerical error 2.6ร against a baseline FP8 attention, and its own paper notes FP8 is not always faster than BF16 for every head size and masking setup.
10. Beyond NVIDIA
Kernel tools ร hardware
Our reading of project documentation. Tap a cell.
Kernels are a large part of NVIDIA's moat. A decade of cuBLAS, cuDNN, CUTLASS and community kernels means new model ideas usually run fastest on NVIDIA first. AMD answers with ROCm, its Composable Kernel and hipBLASLt libraries, and Triton support, so code written in Triton can often move across. Google's TPUs mostly rely on XLA, with Pallas (lowered through Mosaic) for hand-written kernels. AWS offers NKI for Trainium. Each alternative chip must win not only on price per FLOP but on how quickly the important kernels reach high utilisation.
11. Who writes kernels
12. Can AI write kernels?
Because kernels are short, measurable and valuable, they are a natural target for AI coding systems. Stanford's KernelBench (ICML 2025) gives models 250 PyTorch workloads and measures fastp: the share of generated kernels that are correct and faster than PyTorch by a factor p. Its finding: frontier reasoning models did best but matched the PyTorch baseline in fewer than 20% of cases out of the box, improving with execution and profiling feedback. Later experiments reported individual AI-generated kernels well ahead of PyTorch (for example a 4.8ร faster layer norm โ ๏ธ, secondary coverage), and benchmarks have extended to multi-GPU kernels. Correctness checking is the hard part: several reported speedups have come from kernels that exploited loopholes in the test rather than doing the work faster.
13. Choosing
Which tool for your kernel?
An editorial guide.
14. Reading the numbers
- Primary
- FA3: 740 TFLOPS BF16 (75%), ~1.2 PFLOPS FP8, 1.5โ2.0ร over FA2, 2.6ร lower FP8 error (paper, arXiv 2407.08608). FA2 ~35% on H100 (FA3 paper). FA4: up to 1,613 TFLOPS on B200 BF16 (~71%), 1.3ร cuDNN 9.13, 2.7ร Triton, CuTe-DSL with 20โ30ร faster compiles (arXiv 2603.05451; some summaries say 1,605). KernelBench: 250 tasks, <20% matched PyTorch (paper).
- Repository / secondary โ ๏ธ
- DeepGEMM up to ~1,550 TFLOPS on H800 (repo; press often cites 1,350+). ThunderKittens ~30% faster than FA2 on attention (launch coverage). AI-written 4.8ร layer norm (press). NVFP4 and NKI details from vendor material.
- Chip figures
- Dense datasheet peaks, rounded: A100 312 TFLOPS BF16, 2.0 TB/s; H100 SXM ~989, 3.35 TB/s; B200 ~2,250, ~8 TB/s. Real kernels rarely exceed 70โ80%.
- Our illustrations
- Roofline intensities, memory ladder bandwidths, tiling counts and fusion traffic are simplified models.
- Papers
- FlashAttention (2022)
- FlashAttention-2 (2023)
- FlashAttention-3 (2024)
- FlashAttention-4 (2026)
- KernelBench
- Roofline model (Williams et al., 2009)
- Projects and posts
- OpenAI: Introducing Triton
- NVIDIA CUTLASS and CuTe
- DeepGEMM ยท FlashMLA
- Hazy Research: GPUs go brrr / ThunderKittens
- JAX Pallas documentation
- Together AI: FlashAttention-3
- Part 3: parallelism libraries ยท Part 2: PyTorch vs JAX