AI Chips & Compute Infrastructure:
The Physical Foundation of Intelligence

Every post in this series has quietly leaned on one assumption: that compute is available, if you can pay for it. Training a frontier model "costs $100M." Running inference "costs $0.003 per image." A robot foundation model needs "$10-100M in data collection and compute." This post unpacks what's actually behind those numbers — the chips, the data centers, the power grids, and the supply chains that make modern AI physically possible. Algorithms get the headlines, but for the last decade, the single best predictor of a lab's frontier capability has been the compute it can put behind a training run. This is the layer everything else in this blog sits on top of.

🕰️ From CPUs to AI Factories

The hardware story runs in parallel to every model story we've covered — each capability leap in this blog series was gated by a corresponding leap in silicon. Here's how we got from general-purpose chips to purpose-built AI infrastructure:

📊 Why Compute Is the Bottleneck

Scaling laws research (Kaplan et al. 2020, Chinchilla/Hoffmann et al. 2022) established an empirical relationship that has held remarkably well: model performance improves predictably as a function of compute, data, and parameters scaled together. This turned AI research from a purely algorithmic pursuit into something closer to a capital-intensive industrial process — the labs with the most compute, deployed most efficiently, have consistently produced the most capable frontier models. Algorithmic efficiency gains (better architectures, training recipes, data curation) have been real and valuable, but they've reduced the compute needed for a given capability level rather than eliminating the advantage of having more compute in absolute terms. This is why frontier labs' single largest expense — and the reason hyperscalers now spend more on AI infrastructure CapEx than on nearly anything else in their business — is compute.

🧩 A Taxonomy of AI Silicon

Not all "AI chips" are alike — they diverge sharply based on whether they're optimized for training or inference, general flexibility or maximum efficiency on a narrow workload, and data-center scale or edge deployment.

By Design Philosophy

By Workload Target

🗂️ The Chip Landscape

⚙️ How AI Chips Actually Work

Understanding a few core architectural concepts explains why chip specs and pricing look the way they do — and why "more FLOPs" alone doesn't determine which chip wins a given workload.

1. Precision: The FP32 → FP4 Journey

Neural networks don't need the full numerical precision that scientific computing traditionally demanded. Each step down in precision roughly doubles throughput and halves memory footprint, at some cost to numerical accuracy that modern training/inference techniques increasingly compensate for algorithmically.

2. Memory Bandwidth: The Real Bottleneck

Raw FLOPs get the marketing headlines, but for most modern AI workloads — especially LLM inference — the binding constraint is memory bandwidth, not compute. Generating each output token requires streaming the entire model's weights from memory to the compute units; if memory can't feed data fast enough, expensive compute cores sit idle waiting. This is why High Bandwidth Memory (HBM) capacity and speed are now a headline spec competitive battleground in their own right — AMD's MI300X leads on raw HBM capacity specifically to win memory-bound inference workloads away from NVIDIA.

3. Interconnect: Making Thousands of Chips Act as One

No single chip trains a frontier model — modern training runs span thousands to tens of thousands of accelerators working in tight synchronization. NVLink connects GPUs within a server/rack at extremely high bandwidth (Blackwell's NVL72 links 72 GPUs into one coherent domain); InfiniBand or high-speed Ethernet then connects racks to each other across the data center. The efficiency of this interconnect fabric — not just individual chip speed — determines how much of theoretical FLOPs a training run actually achieves; poor interconnect utilization is why real-world training throughput is often only 30-50% of a cluster's theoretical peak.

4. Chiplets & Advanced Packaging

As single monolithic dies approach the physical reticle limit (the maximum size a chip fabrication process can print in one pass), the industry has shifted to chiplet designs — multiple smaller dies fused together via advanced packaging (like TSMC's CoWoS) into what functions as one larger logical chip. Blackwell's two-die fusion and AMD's MI300X chiplet architecture are both direct responses to this physical scaling limit — packaging technology, not just transistor design, has become a genuine competitive bottleneck.

🏢 The Data Center Layer: From Chip to "AI Factory"

A chip alone trains nothing — it has to sit inside a data center engineered to deliver enough power and remove enough heat to let thousands of them run at full tilt simultaneously. NVIDIA's own framing has shifted to calling these facilities "AI factories," reflecting that the unit of production and competition has moved up a level from individual chips.

The most-cited industry rule of thumb by 2026: a single rack of the latest-generation GPU hardware can draw well over 100kW — more power than an entire typical single-family home's annual usage, running continuously, in one server rack. Multiply that across a data center with tens of thousands of chips and it becomes clear why power procurement, not chip acquisition, is increasingly the binding constraint on how fast a lab can grow its training capacity.

🏭 Major Players

🚀 Startups & Funding Activity

Chip startups face a brutal capital reality — designing and fabricating leading-edge silicon requires hundreds of millions before any revenue, and success depends on out-executing NVIDIA's software ecosystem, not just matching its hardware specs. Despite this, several companies have raised eye-catching rounds betting on architectural approaches NVIDIA hasn't prioritized.

Investor thesis patterns: Cerebras and SambaNova bet on radical departures from conventional GPU architecture (wafer-scale integration, dataflow processors) targeting workloads where NVIDIA's general-purpose flexibility is a liability, not an asset. Groq and Etched bet narrowly on inference-only, transformer-specific silicon, trading training flexibility for maximum inference efficiency on the workload that will dominate long-term AI compute spend as deployed usage outstrips training runs. Meta's 2024 acqui-hire of Rivos signals — like Amazon's Covariant acquisition in robotics — that hyperscalers view chip design talent as strategically scarce enough to acquire outright.

📐 How AI Chips Are Benchmarked

Peak theoretical FLOPs figures on a spec sheet are notoriously misleading — real-world training and inference throughput depends heavily on memory bandwidth, interconnect efficiency, and software maturity. Standardized benchmarks exist specifically to cut through marketing numbers.

💰 Compute Economics

The connection to model pricing: Every API price-per-token figure discussed in our Frontier Model Pricing post ultimately decomposes into this hardware cost stack — chip acquisition/depreciation, power, cooling, networking, and data center real estate, divided by achieved tokens-per-second-per-dollar at a given precision and batch size. As chip efficiency improves roughly in line with new generations (Blackwell targets meaningfully lower cost-per-token than Hopper), API prices for comparable model capability have fallen accordingly — the single biggest driver of the "price collapse" trend documented in that earlier post is this underlying hardware efficiency curve, not just competitive pressure.

🔬 Research Frontiers

🌍 Geopolitics & the Energy Constraint

Compute infrastructure has become inseparable from national industrial policy and energy strategy in a way that would have seemed strange a decade ago.

⚖️ The Verdict: Is the Compute Build-Out Sustainable?