Home โ€บ Blog โ€บ The Memory Wall: Why HBM Decides Who Wins AI
AI Compute Stack ยท Part 9 ยท Memory

The Memory Wall: Why HBM Decides Who Wins AI

For twenty years chips learned to calculate far faster than memory could feed them. Large language models turned that gap into the defining constraint of AI: a GPU generating text spends most of its time waiting for data. This guide explains the memory wall, the stacked memory called HBM that holds it back, the three companies that make it, why it made your next laptop more expensive, and the hardware and software tricks for getting over it.

FL
FrontierAGI Team
โš ๏ธ How this was researched Everything here comes from web searches run up to 23 October 2026. Chip specifications are mostly vendor figures as reported by the technical press; market shares differ between trackers and are shown side by side; price figures for consumer memory vary widely by source. The speed-limit calculator and roofline chart are simplified teaching models, labelled as upper bounds. See Reading the numbers for conflicts.

The AI boom made your laptop more expensive

In 2026 a 32 GB kit of desktop memory that cost around $110 to $140 a year earlier was selling for about $400 in one price tracker. Gartner expects combined DRAM and SSD prices to rise about 130% by the end of 2026, pushing average PC prices up 17% and PC shipments down about 10%. Counterpoint estimates the cost of materials for a smartphone could rise 15% or more. Dell raised PC prices in January.

The cause is not phones or PCs. Memory makers moved factory capacity to a specialised, far more profitable product: high-bandwidth memory (HBM), the stacked DRAM that sits beside every AI accelerator. HBM is reported sold out for 2026, and Bernstein does not expect the shortage to ease before 2027. To understand why AI needs so much of it, you need one idea: the memory wall.

3.0ร— vs 1.6ร—growth every two years in peak compute vs memory bandwidth over 20 years (Gholami et al.)
22 TB/smemory bandwidth of one Nvidia Rubin GPU, about 6.6ร— an H100
63%memory's share of AI chip component costs, up from 52% (Epoch AI)
~$26.5Braised by SK hynix in its July 2026 US listing, largely to expand HBM capacity

What the memory wall is

The term dates to the 1990s, but the clearest recent evidence comes from Amir Gholami and colleagues' paper "AI and Memory Wall" (2024, later in IEEE Micro). Looking at twenty years of server hardware, they found peak compute grew about 3.0ร— every two years, while DRAM bandwidth grew about 1.6ร— and interconnect bandwidth about 1.4ร—. Compound those rates and the gap becomes enormous.

Twenty years of diverging growth

Index, year 0 = 1, log scale. Curves are the paper's average rates compounded, not measured data points. Drag the slider to see the gap after N years.

Put simply: each new chip generation can do far more maths per second, but the pipe that brings it numbers to work on widens much more slowly. If the work needs a lot of data per calculation, the chip idles.

Why large language models hit the wall so hard

Text generation has two phases. Prefill reads your whole prompt at once: lots of maths per byte, so chips stay busy. Decode then produces the answer one token at a time, and for every single token the model must read all of its weights from memory. For one user, that is roughly one calculation for every byte read.

An H100 can do about 989 trillion 16-bit operations a second but read only about 3.35 trillion bytes a second, so it needs about 295 operations per byte to stay busy (Baseten's figure). Decode for a single user offers about one. The chip spends over 99% of its potential waiting. That is why serving systems batch many users together: the same weights read once serve many requests. And it is why, at large batches, the KV cache (each conversation's stored attention state, which grows with context length) becomes the next thing clogging the memory pipe.

๐Ÿ’ก A rule of thumb For a single user, the fastest a model can generate is roughly memory bandwidth รท model size in bytes. A 70-billion-parameter model in 16-bit precision is about 140 GB; on an H100 at 3.35 TB/s that caps decode at about 24 tokens per second, before any other overhead. Faster chips barely help this number. More bandwidth, or fewer bytes, does.

The roofline: when does a GPU stop waiting?

The roofline model draws this as two ceilings. Below a certain number of operations per byte, performance rises in a straight line with memory bandwidth (the sloped roof). Above it, the chip hits its compute limit (the flat roof). Drag the batch size to move a decode workload along the line.

H100 roofline for decode

Uses H100 figures (989 TFLOPS dense BF16, 3.35 TB/s). Simplification: with 16-bit weights, decode intensity โ‰ˆ batch size in operations per byte, ignoring KV-cache reads, which lower it in practice.

Speed-limit calculator

Pick a model size, a precision and a chip to see the single-user decode ceiling from the rule of thumb above.

Maximum tokens per second for one user

An upper bound: bandwidth รท weight bytes. Real systems are slower (KV cache, kernels, multi-GPU communication) and serve many users at once. If the model is bigger than one chip's memory, it must be split across several chips.

Inside HBM

Ordinary DRAM sits on sticks a few centimetres from the processor, connected by 64 wires per channel. HBM takes a radically different approach: stack DRAM dies vertically, drill thousands of vertical connections through them, and place the stack millimetres from the GPU on a shared silicon base. The result is a very wide, very short connection. Click the parts.

Cross-section of an HBM stack beside a GPU

Schematic, not to scale.

HBM3E vs HBM4

From design guides citing the JEDEC standards (HBM3E: JESD238; HBM4: JESD270-4, April 2025). Per-stack figures; products vary.

HBM3EHBM4
Interface width1,024 bits2,048 bits
Bandwidth per stack~1.2โ€“1.33 TB/s~2.0 TB/s, up to ~3.3 TB/s in faster versions
Pin speedLower (the wider HBM4 bus does most of the work)6.4โ€“12.8 Gb/s (Samsung demonstrated ~13)
Max capacity36 GB (12-high)64 GB (16-high, 32 Gb dies)
Compatibilityโ€”Not backward-compatible with HBM3/3E controllers
Base dieMade by the memory makerIncreasingly logic-process dies from TSMC or Samsung Foundry; "custom HBM" from HBM4E

HBM history, 2010 to 2026

Timeline

Click an event.

The chip race is a memory race

Look at what changed between AI chip generations and memory is the headline every time. Nvidia reportedly pushed HBM4 speeds above earlier targets for Rubin, delaying suppliers' production, partly to stay ahead of AMD's MI455X.

Memory per accelerator

Vendor figures as reported. Toggle between capacity and bandwidth.

AMD's 72-GPU Helios rack adds up to more than 31 TB of HBM4 and about 1.7 PB/s of memory bandwidth. Google's Ironwood TPU carries 192 GB across eight HBM3E stacks, with 2.7 times the bandwidth of its predecessor, TPU v5p.

Three companies hold the key

Only SK hynix, Samsung and Micron make HBM at scale. SK hynix, which co-created the standard with AMD, has led since the AI boom began, but estimates of the split differ.

HBM market share by source

Each tab is a different tracker or period; they do not agree, so read them as a range.

Memory is now the expensive part

Epoch AI's analysis of AI chip component costs finds memory's share rising from 52% to 63%, while packaging and other parts shrank. Total component spending on AI chips grew from about $22 billion in 2024 to $52 billion in 2025, with HBM alone accounting for roughly $20 billion of that increase.

Memory's share of AI chip component costs

Source: Epoch AI, AI chip component cost shares. Secondary estimates put HBM3E at about $300 per stack and HBM4 at about $500 (unverified).

The consumer bill

Every wafer turned into HBM is a wafer not turned into ordinary DRAM, and an HBM stack uses several times the silicon of the same capacity of standard memory. As makers reallocated, consumer supply tightened.

What the memory crunch did to prices

Forecasts and reports from different sources and dates; not directly comparable.

Sources: Gartner, IDC, Counterpoint (via Bloomberg and trade press). Bloomberg reports some memory prices more than 10 times their January 2025 levels.

China and the HBM export ban

In December 2024 the United States restricted exports of HBM, the tools to make it and advanced packaging equipment to China, and added more than 140 Chinese chip firms to its Entity List. Because HBM is the memory every frontier AI chip needs, the rules target the bottleneck rather than the chip.

Ways over the wall

There are two broad strategies: build a wider pipe, or send fewer bytes through it. Pick an approach.

Hardware and software fixes

Which fix helps which bottleneck?

Our summary. โœ“ main effect, ~ partial, โœ— little effect. Hover for the reason.

What to watch to 2028

๐Ÿ“ˆ Supply

Whether HBM stays sold out into 2027, how fast SK hynix spends its listing proceeds, Samsung's share of Rubin, and when consumer DRAM prices fall.

๐Ÿงช Technology

HBM4E and custom base dies in 2027, high-bandwidth flash samples and first products, CXL pools in production inference, and whether SRAM inference chips win share after Nvidia's Groq deal.

๐ŸŒ Geopolitics

CXMT's HBM3 yields and any HBM3E confirmation, tighter or looser export rules, and Huawei's stockpile running down.

Reading the numbers: where sources disagree

ClaimWhat the sources say
"60,000ร— compute vs 100ร— bandwidth"A third-party summary of the Gholami paper; we could only confirm the paper's per-two-year rates (3.0ร—, 1.6ร—, 1.4ร—).
SK hynix's HBM share63.2% of 2025 revenue and 56.4% in Q1 2026 (IDC, via SK hynix's SEC filings); 58% in Q1 2026 (Counterpoint); a Korean report says Samsung has risen to about 33%.
Rubin bandwidth22 TB/s at announcement; 22.2 TB/s after a reported 10% upgrade.
Ironwood bandwidth7.2 or 7.3 TB/s depending on source.
Consumer price risesFrom +89% to +110% for DDR5 in different trackers, more than 10ร— for some parts (Bloomberg); Gartner's +130% is DRAM and SSD combined by end-2026.
DeepSeek's "93.3% smaller KV cache"Compared with DeepSeek 67B, not a general MLA-vs-standard-attention figure.
PagedAttention memory wasteReported as cut from 60โ€“80% to about 4%, or utilisation raised to ~96%; figures differ by write-up.
High-bandwidth flash "8โ€“16ร— capacity at similar cost"Vendor claim; top bandwidth figures are simulations.
CXL pooling "5ร— faster than SSD or RDMA offload"A vendor demonstration; a 2026 preprint argues rack-scale electrical CXL is not yet practical.
CXMT HBM3EReported small-scale production, not confirmed by the company.

๐Ÿ”— References and papers

๐Ÿงญ Closing

๐ŸŽฏ The bottom line AI's limiting resource inside the chip is not maths; it is moving data. Because generating each token means re-reading the whole model, memory bandwidth sets the speed limit, memory capacity sets how many users and how much context fit, and memory now makes up most of an accelerator's component cost. That is why three memory makers have become as strategic as Nvidia, why export controls target HBM, and why your next laptop costs more. The way over the wall is both a wider pipe (HBM4, custom base dies, high-bandwidth flash, SRAM chips) and fewer bytes (batching, quantization, smarter KV caches). Watch the memory roadmap as closely as the GPU roadmap.