The Memory Wall: Why HBM Decides Who Wins AI
For twenty years chips learned to calculate far faster than memory could feed them. Large language models turned that gap into the defining constraint of AI: a GPU generating text spends most of its time waiting for data. This guide explains the memory wall, the stacked memory called HBM that holds it back, the three companies that make it, why it made your next laptop more expensive, and the hardware and software tricks for getting over it.
The AI boom made your laptop more expensive
In 2026 a 32 GB kit of desktop memory that cost around $110 to $140 a year earlier was selling for about $400 in one price tracker. Gartner expects combined DRAM and SSD prices to rise about 130% by the end of 2026, pushing average PC prices up 17% and PC shipments down about 10%. Counterpoint estimates the cost of materials for a smartphone could rise 15% or more. Dell raised PC prices in January.
The cause is not phones or PCs. Memory makers moved factory capacity to a specialised, far more profitable product: high-bandwidth memory (HBM), the stacked DRAM that sits beside every AI accelerator. HBM is reported sold out for 2026, and Bernstein does not expect the shortage to ease before 2027. To understand why AI needs so much of it, you need one idea: the memory wall.
What the memory wall is
The term dates to the 1990s, but the clearest recent evidence comes from Amir Gholami and colleagues' paper "AI and Memory Wall" (2024, later in IEEE Micro). Looking at twenty years of server hardware, they found peak compute grew about 3.0ร every two years, while DRAM bandwidth grew about 1.6ร and interconnect bandwidth about 1.4ร. Compound those rates and the gap becomes enormous.
Twenty years of diverging growth
Index, year 0 = 1, log scale. Curves are the paper's average rates compounded, not measured data points. Drag the slider to see the gap after N years.
Put simply: each new chip generation can do far more maths per second, but the pipe that brings it numbers to work on widens much more slowly. If the work needs a lot of data per calculation, the chip idles.
Why large language models hit the wall so hard
Text generation has two phases. Prefill reads your whole prompt at once: lots of maths per byte, so chips stay busy. Decode then produces the answer one token at a time, and for every single token the model must read all of its weights from memory. For one user, that is roughly one calculation for every byte read.
An H100 can do about 989 trillion 16-bit operations a second but read only about 3.35 trillion bytes a second, so it needs about 295 operations per byte to stay busy (Baseten's figure). Decode for a single user offers about one. The chip spends over 99% of its potential waiting. That is why serving systems batch many users together: the same weights read once serve many requests. And it is why, at large batches, the KV cache (each conversation's stored attention state, which grows with context length) becomes the next thing clogging the memory pipe.
The roofline: when does a GPU stop waiting?
The roofline model draws this as two ceilings. Below a certain number of operations per byte, performance rises in a straight line with memory bandwidth (the sloped roof). Above it, the chip hits its compute limit (the flat roof). Drag the batch size to move a decode workload along the line.
H100 roofline for decode
Uses H100 figures (989 TFLOPS dense BF16, 3.35 TB/s). Simplification: with 16-bit weights, decode intensity โ batch size in operations per byte, ignoring KV-cache reads, which lower it in practice.
Speed-limit calculator
Pick a model size, a precision and a chip to see the single-user decode ceiling from the rule of thumb above.
Maximum tokens per second for one user
An upper bound: bandwidth รท weight bytes. Real systems are slower (KV cache, kernels, multi-GPU communication) and serve many users at once. If the model is bigger than one chip's memory, it must be split across several chips.
Inside HBM
Ordinary DRAM sits on sticks a few centimetres from the processor, connected by 64 wires per channel. HBM takes a radically different approach: stack DRAM dies vertically, drill thousands of vertical connections through them, and place the stack millimetres from the GPU on a shared silicon base. The result is a very wide, very short connection. Click the parts.
Cross-section of an HBM stack beside a GPU
Schematic, not to scale.
HBM3E vs HBM4
From design guides citing the JEDEC standards (HBM3E: JESD238; HBM4: JESD270-4, April 2025). Per-stack figures; products vary.
| HBM3E | HBM4 | |
|---|---|---|
| Interface width | 1,024 bits | 2,048 bits |
| Bandwidth per stack | ~1.2โ1.33 TB/s | ~2.0 TB/s, up to ~3.3 TB/s in faster versions |
| Pin speed | Lower (the wider HBM4 bus does most of the work) | 6.4โ12.8 Gb/s (Samsung demonstrated ~13) |
| Max capacity | 36 GB (12-high) | 64 GB (16-high, 32 Gb dies) |
| Compatibility | โ | Not backward-compatible with HBM3/3E controllers |
| Base die | Made by the memory maker | Increasingly logic-process dies from TSMC or Samsung Foundry; "custom HBM" from HBM4E |
HBM history, 2010 to 2026
Timeline
Click an event.
The chip race is a memory race
Look at what changed between AI chip generations and memory is the headline every time. Nvidia reportedly pushed HBM4 speeds above earlier targets for Rubin, delaying suppliers' production, partly to stay ahead of AMD's MI455X.
Memory per accelerator
Vendor figures as reported. Toggle between capacity and bandwidth.
AMD's 72-GPU Helios rack adds up to more than 31 TB of HBM4 and about 1.7 PB/s of memory bandwidth. Google's Ironwood TPU carries 192 GB across eight HBM3E stacks, with 2.7 times the bandwidth of its predecessor, TPU v5p.
Three companies hold the key
Only SK hynix, Samsung and Micron make HBM at scale. SK hynix, which co-created the standard with AMD, has led since the AI boom began, but estimates of the split differ.
HBM market share by source
Each tab is a different tracker or period; they do not agree, so read them as a range.
- Vera Rubin supply: Jensen Huang said on 5 June 2026 that all three makers had qualified for Rubin's HBM4. Nvidia has not disclosed allocations; analysts estimate SK hynix at 60โ70%, Samsung 25โ30% and Micron the rest. Samsung began HBM4 mass production in February 2026.
- SK hynix goes to Wall Street: in July 2026 it listed American depositary shares at $149 each, raising about $26.5 billion, one of the largest equity offerings ever, with proceeds earmarked for HBM capacity and new fabs.
- Next: custom HBM. HBM4E is expected in 2027 and HBM5 around 2029. Customers will be able to order base dies with their own logic (memory controllers, interfaces), made on TSMC or Samsung logic processes, which ties memory makers more tightly to individual chip designers.
Memory is now the expensive part
Epoch AI's analysis of AI chip component costs finds memory's share rising from 52% to 63%, while packaging and other parts shrank. Total component spending on AI chips grew from about $22 billion in 2024 to $52 billion in 2025, with HBM alone accounting for roughly $20 billion of that increase.
Memory's share of AI chip component costs
The consumer bill
Every wafer turned into HBM is a wafer not turned into ordinary DRAM, and an HBM stack uses several times the silicon of the same capacity of standard memory. As makers reallocated, consumer supply tightened.
What the memory crunch did to prices
Forecasts and reports from different sources and dates; not directly comparable.
China and the HBM export ban
In December 2024 the United States restricted exports of HBM, the tools to make it and advanced packaging equipment to China, and added more than 140 Chinese chip firms to its Entity List. Because HBM is the memory every frontier AI chip needs, the rules target the bottleneck rather than the chip.
- CXMT, China's leading DRAM maker, targets HBM3 mass production by the end of 2026, with early yields reported around 25% for 8-high stacks; small-scale HBM3E production was reported in August 2026 but not confirmed by the company. Estimates put it one to two generations, or three to four years, behind.
- Huawei has relied mainly on HBM stockpiled from Samsung (and to a lesser extent SK hynix) and announced its own "proprietary" HBM in September 2025.
- One estimate projects about 7 million mostly HBM3 dies from China in 2026; its author calls the figure highly uncertain.
Ways over the wall
There are two broad strategies: build a wider pipe, or send fewer bytes through it. Pick an approach.
Hardware and software fixes
Which fix helps which bottleneck?
Our summary. โ main effect, ~ partial, โ little effect. Hover for the reason.
What to watch to 2028
๐ Supply
Whether HBM stays sold out into 2027, how fast SK hynix spends its listing proceeds, Samsung's share of Rubin, and when consumer DRAM prices fall.๐งช Technology
HBM4E and custom base dies in 2027, high-bandwidth flash samples and first products, CXL pools in production inference, and whether SRAM inference chips win share after Nvidia's Groq deal.๐ Geopolitics
CXMT's HBM3 yields and any HBM3E confirmation, tighter or looser export rules, and Huawei's stockpile running down.Reading the numbers: where sources disagree
| Claim | What the sources say |
|---|---|
| "60,000ร compute vs 100ร bandwidth" | A third-party summary of the Gholami paper; we could only confirm the paper's per-two-year rates (3.0ร, 1.6ร, 1.4ร). |
| SK hynix's HBM share | 63.2% of 2025 revenue and 56.4% in Q1 2026 (IDC, via SK hynix's SEC filings); 58% in Q1 2026 (Counterpoint); a Korean report says Samsung has risen to about 33%. |
| Rubin bandwidth | 22 TB/s at announcement; 22.2 TB/s after a reported 10% upgrade. |
| Ironwood bandwidth | 7.2 or 7.3 TB/s depending on source. |
| Consumer price rises | From +89% to +110% for DDR5 in different trackers, more than 10ร for some parts (Bloomberg); Gartner's +130% is DRAM and SSD combined by end-2026. |
| DeepSeek's "93.3% smaller KV cache" | Compared with DeepSeek 67B, not a general MLA-vs-standard-attention figure. |
| PagedAttention memory waste | Reported as cut from 60โ80% to about 4%, or utilisation raised to ~96%; figures differ by write-up. |
| High-bandwidth flash "8โ16ร capacity at similar cost" | Vendor claim; top bandwidth figures are simulations. |
| CXL pooling "5ร faster than SSD or RDMA offload" | A vendor demonstration; a 2026 preprint argues rack-scale electrical CXL is not yet practical. |
| CXMT HBM3E | Reported small-scale production, not confirmed by the company. |
๐ References and papers
- Papers
- Gholami et al., AI and Memory Wall (arXiv 2403.14123)
- DeepSeek-V2 and Multi-head Latent Attention (arXiv 2405.04434)
- vAttention: dynamic memory management without PagedAttention (arXiv 2405.04437)
- LLM Inference Unveiled: survey and roofline insights (arXiv 2402.16363)
- A Photonic-CXL memory appliance for KV cache management (arXiv 2607.27187)
- HPIM: processing-in-memory accelerator for LLM inference (arXiv 2509.12993)
- Explainers on inference and rooflines
- Baseten: calculating inference bottlenecks
- Why LLM decoding is memory-bound
- Hardware and HBM
- Wikipedia: High Bandwidth Memory
- Siemens: HBM3E and HBM4 design guide ยท Rambus on HBM
- Guru3D: Nvidia Rubin with HBM4 ยท Tom's Hardware: Nvidia pushes HBM4 specs higher
- StorageReview: AMD MI455X and Helios
- The Next Platform: Google Ironwood
- TrendForce: HBM4E outlook
- Market, supply and prices
- Epoch AI: AI chip component cost shares
- Nvidia certifies all three suppliers for Rubin HBM4
- Financial News: SK hynix US listing priced
- Bloomberg: why the AI memory shortage makes technology more expensive
- Wikipedia: 2024โpresent global memory supply shortage
- China and export controls
- ChinaTalk: mapping China's HBM advances
- AI Frontiers: HBM and the gaps in US export controls
- New memory approaches
- HPCwire: what is high-bandwidth flash?
- HPCwire: SRAM chips pulling ahead
- The Register: why Nvidia paid $20B for Groq's technology
- Samsung white paper: KV cache offloading to a CXL memory pool
- On this site (AI Compute Stack)
- Part 1: The Power Bottleneck
- Part 2: Inside the AI Factory
- Part 3: AI chips and compute infrastructure
- How LLM inference works