Home โ€บ Blog โ€บ Kernels that matter
Inside the ML Layer ยท Part 4 ยท Kernels

The Kernels That Matter: FlashAttention, CUTLASS, Triton and the Code That Actually Runs on the GPU

Frameworks describe the maths and parallelism libraries split it across thousands of chips, but the arithmetic itself happens inside kernels: small programs launched across tens of thousands of GPU threads at once. A handful of them, mostly matrix multiplies and attention, account for most of an AI factory's compute. Whether they reach 35% or 75% of a chip's peak decides how many GPUs a lab needs. This part explains why, how the best kernels are written, and who writes them.

FL
FrontierAGI Team
โš ๏ธ How this was researched (as of 27 Oct 2026) Sources are the FlashAttention 1โ€“4 papers, the Triton paper and OpenAI's release post, NVIDIA's CUTLASS documentation and GPU datasheets, DeepSeek's DeepGEMM and FlashMLA repositories, Hazy Research's ThunderKittens posts, JAX's Pallas documentation, the KernelBench paper and the roofline model (Williams et al., 2009). Chip peaks are dense (no sparsity) datasheet figures. Arithmetic intensities in the roofline are rounded illustrations. Code samples are simplified for reading. See Reading the numbers.

1. Case file

โš™๏ธ
The kernel layer, October 2026Layer 7 of 8 in the ML layer map, just above the hardware
H100 SXM peak (dense BF16)~989 TFLOPS ยท 3.35 TB/s HBM
FlashAttention-2 on H100~35% utilisation
FlashAttention-3 on H100up to 740 TFLOPS (75%) BF16 ยท ~1.2 PFLOPS FP8
FlashAttention-4 on B200up to 1,613 TFLOPS (~71%), Mar 2026
DeepGEMM (DeepSeek, FP8)up to ~1,550 TFLOPS on H800 (repo)
KernelBench (LLM-written kernels)Best models beat PyTorch in <20% of tasks out of the box

2. Timeline

GPU kernels for AI, 2007 โ†’ 2026

Tap a milestone. Colour shows the family: NVIDIA libraries, attention, languages, open-source labs.

Tap a milestone.

3. The GPU in one picture

A modern data-centre GPU is about a hundred independent processors (streaming multiprocessors, SMs; 132 on an H100 SXM) sharing a large, slow pool of high-bandwidth memory (HBM). Each SM has tensor cores for matrix maths, a small fast scratchpad (shared memory) and registers. Data gets faster and scarcer the closer it is to the arithmetic, and a kernel's job is to move each number up the ladder as few times as possible.

The memory ladder (H100 SXM)

Tap a level. Capacity and bandwidth are approximate; on-chip bandwidths are aggregate across all SMs and vary with access pattern.

4. Why kernels decide speed: the roofline

Every kernel is limited either by arithmetic or by memory traffic. The deciding number is arithmetic intensity: floating-point operations per byte moved from HBM. Below the chip's "ridge point" (peak FLOPS รท bandwidth, about 295 FLOPs per byte on an H100) a kernel is memory-bound and extra tensor cores do nothing; above it, the kernel is compute-bound. Big matrix multiplies sit far to the right. Softmax, layer norm and token-by-token decoding sit far to the left, which is why they are fused and why inference is so often a memory problem.

Interactive roofline

Pick a GPU and an operation. Logโ€“log axes. Intensities are rounded illustrations for typical LLM shapes in BF16.

5. Matrix multiply, the workhorse

Most of a transformer's FLOPs are matrix multiplies (GEMMs): the attention projections and the feed-forward layers. The trick that makes them fast is tiling: load a block of each input into shared memory once, and reuse it for many multiply-adds before fetching the next block. Larger tiles mean more reuse and fewer trips to HBM, up to the limit of on-chip memory.

Tiling animator

C = A ร— B with 8 ร— 8 matrices. Each step computes one output tile. Count how many input numbers are fetched from HBM. Our own illustration.

A
B
C

On NVIDIA hardware, three layers of tools produce GEMMs. cuBLAS and cuBLASLt are NVIDIA's closed, pre-tuned libraries that frameworks call by default. CUTLASS (2017) is NVIDIA's open-source C++ template library exposing the same building blocks, and CuTe is its layout algebra for describing how tiles map onto threads and memory. Hopper (H100) added hardware that the best kernels must use: the Tensor Memory Accelerator (TMA) copies tiles between HBM and shared memory asynchronously, warpgroup MMA (wgmma) instructions feed the tensor cores from shared memory, and warp specialisation dedicates some threads to loading data while others compute. Blackwell adds tensor memory and two-SM cooperative MMA. Each generation makes older kernels leave more performance on the table.

6. FlashAttention, step by step

Standard attention computes a score for every pair of tokens: an N ร— N matrix, written to HBM, read back for the softmax, written again, read again to multiply by V. At long sequence lengths that matrix dominates both memory and time. FlashAttention (Tri Dao and colleagues, 2022) never materialises it. It loads blocks of queries, keys and values into shared memory, computes partial scores, and keeps a running maximum and sum so the softmax can be corrected as each new block arrives (the "online softmax"). The result is mathematically exact, uses memory proportional to N instead of Nยฒ, and reads HBM far less.

Standard vs Flash attention

Step through each algorithm, then move the sequence-length slider to see memory for the score matrix. One head, BF16. Illustration.

Four generations

Peak forward-pass figures reported by each paper on the GPU it targeted. Different chips; read as progress, not a like-for-like race.

FlashAttention-2 ~35% on H100 and FlashAttention-3 740 TFLOPS (75%) are from the FA3 paper; FA3 FP8 "close to 1.2 PFLOPS". FA4: up to 1,613 TFLOPS BF16 on B200 (~71%), 1.3ร— cuDNN 9.13 and 2.7ร— Triton, per the FA4 paper.

7. Fusion

Three kernels or one?

y = dropout(gelu(x + bias)) over a 4,096 ร— 16,384 BF16 activation (64M elements, 128 MB). Count HBM traffic. Our arithmetic, ignoring caches.

Memory-bound operations gain almost nothing from faster arithmetic, but everything from not touching HBM. Fusing them into one kernel, or into the epilogue of the preceding matrix multiply, is the cheapest speedup in the stack. Compilers now do much of this automatically: TorchInductor (Part 2) generates fused Triton kernels and XLA fuses whole programs. Hand-written fused kernels remain common for the hottest paths: attention, normalisation plus quantisation, mixture-of-experts routing and the inference engines' paged attention (Part 7).

8. Ways to write kernels

The same fused row-softmax, three ways

Simplified; real kernels add masking, tiling across long rows and launch code.


      
ToolLevelWhoBest for
CUDA C++Lowest: threads, warps, PTXNVIDIAMaximum control; every NVIDIA feature first
CUTLASS / CuTe (C++)Templates over CUDANVIDIA, open sourcePeak GEMMs and attention; FA3, many vendor kernels
CuTe-DSL (Python)Python front-end to CuTeNVIDIAThe same control with fast compiles; FA4
TritonPython, block-levelOpenAI, open sourceFast to write, good performance; torch.compile backend; runs on NVIDIA and AMD
ThunderKittensSmall embedded DSL in CUDA, tile-levelStanford Hazy ResearchSimple, fast attention-style kernels (May 2024)
PallasPython, inside JAXGoogleCustom kernels on TPUs (via Mosaic) and GPUs
NKI โš ๏ธPythonAWSCustom kernels on Trainium

9. Low precision

Fewer bits, more FLOPS

Dense tensor-core peaks per GPU from NVIDIA datasheets, approximate. Lower precision roughly doubles throughput each step but needs careful scaling to keep accuracy.

Lower precision is now a kernel problem as much as a hardware one. FP8 has a tiny range, so values are stored with scaling factors; how finely those scales are applied decides accuracy. DeepSeek's DeepGEMM (open-sourced Feb 2025) uses fine-grained, block-wise FP8 scaling and promotes partial sums to higher precision on CUDA cores to limit error, reaching up to about 1,550 TFLOPS on an H800 according to its repository. Blackwell adds 4-bit formats (FP4, and NVIDIA's NVFP4 with small-block scaling โš ๏ธ), mostly used for inference so far. FlashAttention-3's FP8 path illustrates the trade-off: it reduced numerical error 2.6ร— against a baseline FP8 attention, and its own paper notes FP8 is not always faster than BF16 for every head size and masking setup.

10. Beyond NVIDIA

Kernel tools ร— hardware

Our reading of project documentation. Tap a cell.

Tap a cell.

Kernels are a large part of NVIDIA's moat. A decade of cuBLAS, cuDNN, CUTLASS and community kernels means new model ideas usually run fastest on NVIDIA first. AMD answers with ROCm, its Composable Kernel and hipBLASLt libraries, and Triton support, so code written in Triton can often move across. Google's TPUs mostly rely on XLA, with Pallas (lowered through Mosaic) for hand-written kernels. AWS offers NKI for Trainium. Each alternative chip must win not only on price per FLOP but on how quickly the important kernels reach high utilisation.

11. Who writes kernels

Tri Dao and collaboratorsFlashAttention 1โ€“4; Princeton and Together AI, with Meta, NVIDIA and Colfax on FA4.
NVIDIAcuBLAS, cuDNN, CUTLASS, CuTe-DSL, Transformer Engine, TensorRT-LLM kernels.
DeepSeekDeepGEMM, FlashMLA (attention for its latent-attention design) and DeepEP, open-sourced Feb 2025.
Stanford Hazy ResearchThunderKittens; research on simple kernel abstractions.
PyTorch and OpenAITriton and TorchInductor, which write most fused kernels automatically.
Every frontier labIn-house performance teams; kernel engineers are among the most sought-after hires (see Who Runs the AI Factory).

12. Can AI write kernels?

Because kernels are short, measurable and valuable, they are a natural target for AI coding systems. Stanford's KernelBench (ICML 2025) gives models 250 PyTorch workloads and measures fastp: the share of generated kernels that are correct and faster than PyTorch by a factor p. Its finding: frontier reasoning models did best but matched the PyTorch baseline in fewer than 20% of cases out of the box, improving with execution and profiling feedback. Later experiments reported individual AI-generated kernels well ahead of PyTorch (for example a 4.8ร— faster layer norm โš ๏ธ, secondary coverage), and benchmarks have extended to multi-GPU kernels. Correctness checking is the hard part: several reported speedups have come from kernels that exploited loopholes in the test rather than doing the work faster.

13. Choosing

Which tool for your kernel?

An editorial guide.

VerdictKernels are where the theoretical FLOPS of an AI factory turn into real ones. The jump from FlashAttention-2 (~35%) to FlashAttention-3 (~75%) on the same H100 is, in effect, doubling the attention throughput of every H100 that adopts it, without buying a chip. Most teams should not write kernels: use the compiler, cuBLAS, FlashAttention and the open libraries from DeepSeek and others. Write your own only for a new architecture or a hot path, in Triton first, and reach for CUTLASS/CuTe-DSL when the last 20% matters.

14. Reading the numbers

Primary
FA3: 740 TFLOPS BF16 (75%), ~1.2 PFLOPS FP8, 1.5โ€“2.0ร— over FA2, 2.6ร— lower FP8 error (paper, arXiv 2407.08608). FA2 ~35% on H100 (FA3 paper). FA4: up to 1,613 TFLOPS on B200 BF16 (~71%), 1.3ร— cuDNN 9.13, 2.7ร— Triton, CuTe-DSL with 20โ€“30ร— faster compiles (arXiv 2603.05451; some summaries say 1,605). KernelBench: 250 tasks, <20% matched PyTorch (paper).
Repository / secondary โš ๏ธ
DeepGEMM up to ~1,550 TFLOPS on H800 (repo; press often cites 1,350+). ThunderKittens ~30% faster than FA2 on attention (launch coverage). AI-written 4.8ร— layer norm (press). NVFP4 and NKI details from vendor material.
Chip figures
Dense datasheet peaks, rounded: A100 312 TFLOPS BF16, 2.0 TB/s; H100 SXM ~989, 3.35 TB/s; B200 ~2,250, ~8 TB/s. Real kernels rarely exceed 70โ€“80%.
Our illustrations
Roofline intensities, memory ladder bandwidths, tiling counts and fusion traffic are simplified models.