The Interconnect Wars: How 100,000 Chips Become One Computer
No single chip can train a frontier model. Tens of thousands must act as one machine, swapping data many times a second, and every microsecond they wait on the network is money burned. That network now earns Nvidia tens of billions a year and has split the industry into rival camps: NVLink against UALink and Ethernet inside the rack, InfiniBand against Ultra Ethernet across the hall, and a race to put lasers right next to the chip. This guide explains each layer, who is fighting for it, and what it means for who wins AI.
โฑ๏ธ Why the network suddenly matters
Training a large model splits the work across thousands of accelerators. Each one computes on its slice, then they all have to exchange results before the next step can start: gradients are summed across copies of the model (data parallelism), layers split across chips trade activations (tensor and pipeline parallelism), and mixture-of-experts models route every token to experts living on other chips. If the network is slow, the fastest chip in the world sits idle waiting for its neighbours.
That is why the interconnect has become a business of its own. Financial-news summaries of Nvidia's results put its networking revenue at about $7.3 billion in Q2 of fiscal 2026, roughly $31 billion for that whole fiscal year (+142%), and $14.8 billion in Q1 of fiscal 2027, up 199% year on year. That one quarter is larger than most chip companies' annual revenue. Arista, the main independent Ethernet switch maker, has raised its 2026 AI-networking target several times, to around $3.5 billion.
๐๏ธ Try it: how network speed decides GPU utilisation
One training step = compute + the communication that cannot be hidden behind it. Move the sliders to see how much of each step the chips spend waiting. Teaching model with illustrative numbers.
Compute time is fixed at 1 second per step. Bandwidths: 100 Gb/s โ 12.5 GB/s; 400 Gb/s โ 50 GB/s; 800 Gb/s โ 100 GB/s; NVLink 4 = 900 GB/s; NVLink 5 = 1.8 TB/s; NVLink 6 = 3.6 TB/s. Real jobs mix several networks at once.
๐ช The three tiers: scale-up, scale-out, scale-across
"The network" is really three different networks stacked on top of each other, each with its own physics, protocols and winners. The closer to the chip, the faster and shorter-reach it is, and the more it behaves like memory rather than like a network.
๐ Zoom out: from one chip to many data centres
Pick a level. Bandwidths are per accelerator and approximate; latencies are orders of magnitude.
๐ Inside the rack: NVLink and the NVL72
Nvidia introduced NVLink in 2016 to connect GPUs faster than PCIe could, and in 2018 added NVSwitch, a switch chip that lets every GPU in a box talk to every other GPU at full speed. The GB200 NVL72 (announced March 2024) pushed that domain out to a whole rack: 72 Blackwell GPUs joined by NVSwitch trays and thousands of copper cables in a "spine" at the back, behaving to software like one enormous GPU. Third-party analyses put it at 1.8 TB/s per GPU and about 130 TB/s for the rack. Nvidia's own NVLink page says sixth-generation NVLink on Vera Rubin doubles that to 3.6 TB/s per GPU and 260 TB/s per 72-GPU rack.
๐ Scale-up bandwidth per GPU, by generation
NVLink bandwidth per GPU (both directions combined), compared with one 800 Gb/s scale-out port. Linear scale.
NVLink 2/3/4: Nvidia V100, A100, H100 specifications. NVLink 5: third-party GB200 analyses. NVLink 6: Nvidia NVLink page. 800 Gb/s port shown as 200 GB/s bidirectional for a like-for-like comparison.
Why copper? Inside a rack the distances are a metre or two, and copper needs no lasers, no transceivers and almost no extra power. The NVL72 spine is a deliberate bet that copper is good enough for scale-up, which is also why the rack has to be so dense (and why the AI factory article spends so much time on 100+ kW racks and liquid cooling). Past a few metres at these speeds, copper runs out and optics take over.
This is Nvidia's strongest moat. Next Platform estimates NVLink/NVSwitch takes roughly three-quarters of all scale-up networking revenue across GPUs and custom chips, and expects that to drop only a few points through 2027. Rivals can match a GPU's FLOPS on paper; matching a 72-chip memory-coherent domain, with the software that assumes it, is much harder.
โ๏ธ The open challengers: UALink, ESUN and NVLink Fusion
Everyone who is not Nvidia (AMD, the hyperscalers designing custom chips, switch makers) needs a scale-up fabric of their own. By late 2026 that need had produced three routes, which Arista summarises as NVLink Fusion, UALink and Ethernet scale-up (ESUN plus SUE-T).
๐บ๏ธ The camps: who backs what
Click a camp.
Sources: UALink Consortium, OCP/ESUN reporting, Next Platform, HPCwire, SemiEngineering, Arista commentary. Product dates conflict between sources.
๐๏ธ Scale-out: InfiniBand vs Ethernet
Beyond the rack, tens of thousands of GPUs are joined by a "back-end" network of NICs (one per GPU, typically 400 or 800 Gb/s), leaf and spine switches, and optical cables, usually built as a fat tree so that any GPU can reach any other at full speed. Both main technologies use RDMA (remote direct memory access): a NIC writes data straight into another machine's memory without bothering either CPU.
- InfiniBand
- Built for supercomputers. Lossless by design (a sender may only transmit when the receiver has buffer space), low latency, with adaptive routing and in-network reductions (SHARP). Effectively a single-vendor technology since Nvidia bought Mellanox in 2019โ2020 for about $6.9 billion. For years the default for AI training.
- Ethernet (RoCE)
- RDMA over Converged Ethernet runs the same idea on ordinary Ethernet. Cheaper, multi-vendor (Broadcom, Arista, Cisco, Nvidia's own Spectrum-X) and familiar to every data-centre team, but classic Ethernet drops packets under congestion and needs careful tuning (priority flow control, congestion control) to behave.
- Ultra Ethernet
- The Ultra Ethernet Consortium (founded 2023) published its 1.0 specification in June 2025 (now v1.0.3, July 2026). It adds packet spraying across many paths, a new transport and modern congestion control, and does not need a lossless fabric. Compliant NICs and switches ramp through 2026โ2027.
Real deployments show both work at enormous scale. Meta built two 24,576-GPU clusters in 2024 with identical GPUs and 400 Gb/s endpoints, one on RoCE (Arista 7800 switches) and one on InfiniBand, and trained Llama 3 on both. xAI's 100,000-GPU Colossus in Memphis runs on Nvidia's Spectrum-X Ethernet; Nvidia says it kept 95% data throughput against about 60% for standard Ethernet (a vendor figure).
๐ฅ Why AI traffic collides
Ordinary data-centre traffic is millions of small, random flows, and classic Ethernet load-balancing (ECMP) hashes each flow onto one path. That works when flows are many and small. AI training is the opposite: a few elephant flows, all starting at the same instant, all needed before anyone can continue. If the hash happens to put two elephants on one link, that link runs at half speed and every GPU waits for the slowest transfer. Packet spraying (in Spectrum-X, Ultra Ethernet and InfiniBand adaptive routing) splits every flow across all paths instead.
๐ฒ Simulator: flow hashing vs packet spraying
8 large flows cross 8 equal paths between two switches. With per-flow hashing, each flow lands on a random path. Press "New collective" to start another training step.
Teaching model: each flow carries one unit of data, each path carries one unit per time step, so a step lasts as long as the busiest path. Real fabrics also face incast at the receiver, retransmissions and stragglers.
๐ช Google's other path: optical circuit switches
Google took a different route for its TPUs. Instead of a packet switch that reads every packet, its Apollo optical circuit switches use arrays of tiny MEMS mirrors to steer light from one fibre straight into another. They do not look at packets at all; they just set up a path, like an old telephone exchange. TPU v4 used them to wire 4,096 chips into a pod whose 3D-torus shape can be changed in software: cubes of 64 chips are joined in different arrangements, faulty cubes are swapped out, and a job gets the topology that suits it. Later generations (Ironwood, TPU 8) scale the same idea to pods of 9,000+ chips, with copper inside a rack and optics between racks.
๐ Rewire a pod with mirrors
Four 64-chip cubes connect through an optical circuit switch. Pick a layout: the mirrors re-point and the same hardware becomes a different machine.
Simplified from Google's TPU v4 paper (Jouppi et al., ISCA 2023) and "Jupiter Evolving" (Poutievski et al., SIGCOMM 2022). TrendForce estimates an OCS uses ~100 W against ~3 kW for an electrical switch of similar role; treat as an estimate.
The trade-off: circuits change in milliseconds to seconds, not per packet, so OCS suits traffic patterns known in advance (like a training job's collectives) rather than random traffic. It saves power and money, and it made Google one of the few players never dependent on Nvidia's networking.
๐ก Light comes to the chip: co-packaged optics
Every scale-out link today ends in a pluggable transceiver: a module the size of a thumb drive that turns electrical signals into light. A large cluster needs several per GPU, each drawing well over 10 watts, and the electrical path from switch chip to module burns power too. At 100,000+ GPUs, optics become a meaningful share of the data centre's power and a common cause of failures. Three answers are competing:
๐ Relative optics power per bit (vendor claims)
Index: conventional DSP pluggable = 100. Lower is better.
Nvidia claims its silicon-photonics switches are about 3.5ร more power-efficient than pluggables; LPO vendors claim roughly half the power of DSP modules. Real savings depend on reach and design.
- Pluggables (DSP)
- Today's workhorse, shipped in huge volumes by Coherent, Lumentum, InnoLight and others. Easy to replace when one fails.
- Linear-drive (LPO)
- Removes the power-hungry signal processor from the module and lets the switch chip drive the optics directly. Cheaper and cooler, but harder to make work at the highest speeds and longer reaches.
- Co-packaged optics (CPO)
- Puts the optical engines on the same package as the switch chip, so signals travel millimetres rather than centimetres over copper. Broadcom has shipped CPO switches since 2024; its Tomahawk 6 "Davisson" reaches 102.4 Tb/s with sixteen 6.4 Tb/s engines. Nvidia's Quantum-X Photonics InfiniBand switch (115 Tb/s, 144 ports of 800 Gb/s) shipped first, and Dell'Oro reports Spectrum-X Photonics Ethernet switches moving into production in 2026.
๐ Scale-across: one training job, many buildings
A single site can only get so much power (see the power bottleneck). So labs increasingly spread training across several data centres tens or hundreds of kilometres apart. That brings new problems: light in fibre takes about 5 microseconds per kilometre, so a 100 km link adds about a millisecond each way, and long-haul capacity is far more expensive than in-building cables.
Nvidia's answer, Spectrum-XGS (announced at Hot Chips in August 2025), adds distance-aware congestion control and telemetry; Nvidia claims nearly 1.9ร better NCCL collective performance across sites. Arista estimates its own scale-across revenue at about $1.2 billion in 2026. On the software side, training methods that communicate less often (hierarchical or asynchronous gradient exchange) reduce how much traffic must cross the long links.
๐ฐ๏ธ Timeline: how the wars started
Purple = open-standard milestones; teal = Nvidia and hyperscaler milestones.
๐ The scorecard
Interconnects side by side
Simplified. "Status" as of October 2026.
| Technology | Tier | Backers | Scale | Status |
|---|---|---|---|---|
| NVLink + NVSwitch | Scale-up | Nvidia (Fusion opens it to partners such as Marvell) | 72 GPUs per rack today; 3.6 TB/s per GPU (NVLink 6) | Shipping in volume |
| UALink | Scale-up | AMD, Intel, hyperscalers, Cornelis and others | Up to 1,024 accelerators in one switch level | Spec 1.0 (2025) and 2.0 (2026); products 2026โ2027 |
| ESUN / SUE-T | Scale-up | Meta, Microsoft, Broadcom, Arista (OCP) | Ethernet switch scale; slim 4-byte header | ESUN 1.0 approved 2026; early products |
| Google ICI + OCS | Scale-up / pod | Google (TPU only) | Thousands of TPUs per reconfigurable pod | In production since TPU v4 |
| InfiniBand | Scale-out | Nvidia | Tens of thousands of GPUs; 800 Gb/s ports | Mature; losing share to Ethernet |
| Ethernet (RoCE / Spectrum-X) | Scale-out | Nvidia, Broadcom, Arista, Cisco | 100,000+ GPUs (Colossus) | Shipping; growing fastest |
| Ultra Ethernet | Scale-out | UEC members (AMD, Broadcom, Cisco, Meta, Microsoftโฆ) | Designed for millions of endpoints | Spec 1.0.3; silicon ramping 2026โ2027 |
| Spectrum-XGS etc. | Scale-across | Nvidia; Arista and others with own products | Multiple sites, up to hundreds of km | Early deployments (vendor-reported) |
๐ญ What to watch in 2027
- First UALink and ESUN racks in production. AMD's Helios (UALink protocol over Broadcom Tomahawk 6 switches) is the test case: can an open scale-up domain match NVL72 in real training runs?
- Nvidia's scale-up share. If Next Platform's ~75% estimate holds through 2027, NVLink remains the stickiest part of Nvidia's moat, stickier than CUDA.
- Optical scale-up. Copper limits a scale-up domain to roughly a rack. Moving NVLink-class links onto optics would allow domains of hundreds or thousands of GPUs, and is where CPO matters most.
- Ultra Ethernet silicon. Whether compliant NICs and switches from several vendors actually interoperate.
- Multi-site training. Whether the largest 2027 training runs are spread across buildings, and how much long-haul fibre that demands.
๐ข Reading the numbers
- Bits vs bytes, one way vs both. Network ports are quoted in gigabits per second (800 Gb/s = 100 GB/s each way). NVLink is quoted in gigabytes per second for both directions combined. Always convert before comparing.
- Nvidia networking revenue. Figures here come from financial-news summaries; Nvidia's Q2 FY2027 release does not break networking out, so no number is given for that quarter.
- Market shares. The ~75% scale-up share and the Ethernet-vs-InfiniBand split are analyst estimates, and sources disagree on timing.
- Vendor claims. "95% vs 60% throughput" (Colossus), "1.9ร NCCL across sites" (Spectrum-XGS) and "3.5ร better power efficiency" (CPO) are Nvidia's own figures.
- Arista's target appears as $3.25B, $3.5B or $3.6B depending on the source and date.
- Simulators. The utilisation and collision models are deliberately simple and use illustrative numbers.
๐ References and papers
- Papers
- Jouppi et al., "TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning" (ISCA 2023)
- Poutievski et al., "Jupiter Evolving: Transforming Google's Datacenter Network via Optical Circuit Switches" (SIGCOMM 2022)
- Gangidi et al., "RDMA over Ethernet for Distributed AI Training at Meta Scale" (SIGCOMM 2024)
- Llama Team, "The Llama 3 Herd of Models" (infrastructure and network sections)
- DeepSeek-AI, "Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures"
- DeepSeek-AI, "Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning"
- Shoeybi et al., "Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism"
- Standards and vendors
- Nvidia: NVLink and NVLink Switch (NVLink 6 figures)
- Nvidia: Scaling AI factories with co-packaged optics
- Nvidia: Spectrum-X and xAI Colossus
- Nvidia: Q2 fiscal 2027 results
- Meta: Building Meta's GenAI infrastructure (two 24K-GPU clusters)
- IP Infusion: Ultra Ethernet explained (UEC 1.0 vs RoCEv2 and InfiniBand)
- Analysis and reporting
- Next Platform: NVSwitch is the InfiniBand of scale-up networks (Aug 2026)
- SemiEngineering: Multiple AI scale-up options emerge
- HPCwire: Cornelis to build scale-up interconnect with UALink (Sep 2026)
- Locsic: Arista takes Ethernet into the rack
- Dell'Oro: OCP 2026, optics take centre stage
- SemiAnalysis: Co-packaged optics
- TrendForce: Google's optical interconnect and 800G optics (Feb 2026)
- DCD: Going all optical
- Nasdaq/Zacks: Nvidia's networking revenues double
- TIKR: Arista's AI target in 2026
- theCUBE Research: Nvidia scaling AI beyond the walls of the data centre (Spectrum-XGS)
- Related on this site
- AI Chips & Compute Infrastructure ยท The Memory Wall ยท Custom Silicon vs Nvidia ยท Inside the AI Factory