Home โ€บ Blog โ€บ The Interconnect Wars
AI Compute Stack ยท Part 8 ยท Networking

The Interconnect Wars: How 100,000 Chips Become One Computer

No single chip can train a frontier model. Tens of thousands must act as one machine, swapping data many times a second, and every microsecond they wait on the network is money burned. That network now earns Nvidia tens of billions a year and has split the industry into rival camps: NVLink against UALink and Ethernet inside the rack, InfiniBand against Ultra Ethernet across the hall, and a race to put lasers right next to the chip. This guide explains each layer, who is fighting for it, and what it means for who wins AI.

FL
FrontierAGI Team
โš ๏ธ How this was researched This draws on web searches run up to 23 October 2026. NVLink bandwidths for Vera Rubin come from Nvidia's NVLink page. GB200 figures come from third-party analyses, and Nvidia's networking revenue comes from financial-news summaries of its results; Nvidia's Q2 FY2027 release does not break networking out. Market-share figures (scale-up, Ethernet vs InfiniBand) are analyst estimates that conflict, and performance numbers such as "95% throughput" or "1.9ร— faster" are vendor claims. The two simulators are teaching models with illustrative numbers. See Reading the numbers.

โฑ๏ธ Why the network suddenly matters

Training a large model splits the work across thousands of accelerators. Each one computes on its slice, then they all have to exchange results before the next step can start: gradients are summed across copies of the model (data parallelism), layers split across chips trade activations (tensor and pipeline parallelism), and mixture-of-experts models route every token to experts living on other chips. If the network is slow, the fastest chip in the world sits idle waiting for its neighbours.

That is why the interconnect has become a business of its own. Financial-news summaries of Nvidia's results put its networking revenue at about $7.3 billion in Q2 of fiscal 2026, roughly $31 billion for that whole fiscal year (+142%), and $14.8 billion in Q1 of fiscal 2027, up 199% year on year. That one quarter is larger than most chip companies' annual revenue. Arista, the main independent Ethernet switch maker, has raised its 2026 AI-networking target several times, to around $3.5 billion.

~$31BNvidia networking revenue, fiscal 2026 (secondary reporting)
3.6 TB/sNVLink 6 bandwidth per GPU on Vera Rubin (Nvidia)
260 TB/sTotal NVLink bandwidth in one 72-GPU Vera Rubin rack
~75%Nvidia's estimated share of scale-up networking revenue (Next Platform)

๐ŸŽ›๏ธ Try it: how network speed decides GPU utilisation

One training step = compute + the communication that cannot be hidden behind it. Move the sliders to see how much of each step the chips spend waiting. Teaching model with illustrative numbers.

Gradients, activations and expert traffic, in gigabytes
From a 100 Gb/s Ethernet port up to NVLink 6
Good software overlaps sending data with doing maths

Compute time is fixed at 1 second per step. Bandwidths: 100 Gb/s โ‰ˆ 12.5 GB/s; 400 Gb/s โ‰ˆ 50 GB/s; 800 Gb/s โ‰ˆ 100 GB/s; NVLink 4 = 900 GB/s; NVLink 5 = 1.8 TB/s; NVLink 6 = 3.6 TB/s. Real jobs mix several networks at once.

๐Ÿชœ The three tiers: scale-up, scale-out, scale-across

"The network" is really three different networks stacked on top of each other, each with its own physics, protocols and winners. The closer to the chip, the faster and shorter-reach it is, and the more it behaves like memory rather than like a network.

๐Ÿ” Zoom out: from one chip to many data centres

Pick a level. Bandwidths are per accelerator and approximate; latencies are orders of magnitude.

๐Ÿ’ก The key idea Scale-up (inside the rack) lets GPUs read and write each other's memory directly, as if they were one big chip. Scale-out (across racks) sends messages between separate computers. Scale-across (between buildings) stretches that across kilometres. Modern parallelism strategies are designed around this hierarchy: chatty work like tensor parallelism stays inside the scale-up domain; less frequent gradient exchange crosses the scale-out network.

Nvidia introduced NVLink in 2016 to connect GPUs faster than PCIe could, and in 2018 added NVSwitch, a switch chip that lets every GPU in a box talk to every other GPU at full speed. The GB200 NVL72 (announced March 2024) pushed that domain out to a whole rack: 72 Blackwell GPUs joined by NVSwitch trays and thousands of copper cables in a "spine" at the back, behaving to software like one enormous GPU. Third-party analyses put it at 1.8 TB/s per GPU and about 130 TB/s for the rack. Nvidia's own NVLink page says sixth-generation NVLink on Vera Rubin doubles that to 3.6 TB/s per GPU and 260 TB/s per 72-GPU rack.

๐Ÿ“ˆ Scale-up bandwidth per GPU, by generation

NVLink bandwidth per GPU (both directions combined), compared with one 800 Gb/s scale-out port. Linear scale.

NVLink 2/3/4: Nvidia V100, A100, H100 specifications. NVLink 5: third-party GB200 analyses. NVLink 6: Nvidia NVLink page. 800 Gb/s port shown as 200 GB/s bidirectional for a like-for-like comparison.

Why copper? Inside a rack the distances are a metre or two, and copper needs no lasers, no transceivers and almost no extra power. The NVL72 spine is a deliberate bet that copper is good enough for scale-up, which is also why the rack has to be so dense (and why the AI factory article spends so much time on 100+ kW racks and liquid cooling). Past a few metres at these speeds, copper runs out and optics take over.

This is Nvidia's strongest moat. Next Platform estimates NVLink/NVSwitch takes roughly three-quarters of all scale-up networking revenue across GPUs and custom chips, and expects that to drop only a few points through 2027. Rivals can match a GPU's FLOPS on paper; matching a 72-chip memory-coherent domain, with the software that assumes it, is much harder.

โš”๏ธ The open challengers: UALink, ESUN and NVLink Fusion

Everyone who is not Nvidia (AMD, the hyperscalers designing custom chips, switch makers) needs a scale-up fabric of their own. By late 2026 that need had produced three routes, which Arista summarises as NVLink Fusion, UALink and Ethernet scale-up (ESUN plus SUE-T).

๐Ÿ—บ๏ธ The camps: who backs what

Click a camp.

Sources: UALink Consortium, OCP/ESUN reporting, Next Platform, HPCwire, SemiEngineering, Arista commentary. Product dates conflict between sources.

โš ๏ธ Timing is the open question Reports disagree on when UALink products ship: some say late 2026, the consortium points to silicon in labs in the second half of 2026 and products in 2027. ESUN 1.0 was approved in early 2026, but switches built for it are only now appearing. Meanwhile Nvidia ships NVLink in volume today. The challengers are racing a moving target.

๐ŸŸ๏ธ Scale-out: InfiniBand vs Ethernet

Beyond the rack, tens of thousands of GPUs are joined by a "back-end" network of NICs (one per GPU, typically 400 or 800 Gb/s), leaf and spine switches, and optical cables, usually built as a fat tree so that any GPU can reach any other at full speed. Both main technologies use RDMA (remote direct memory access): a NIC writes data straight into another machine's memory without bothering either CPU.

InfiniBand
Built for supercomputers. Lossless by design (a sender may only transmit when the receiver has buffer space), low latency, with adaptive routing and in-network reductions (SHARP). Effectively a single-vendor technology since Nvidia bought Mellanox in 2019โ€“2020 for about $6.9 billion. For years the default for AI training.
Ethernet (RoCE)
RDMA over Converged Ethernet runs the same idea on ordinary Ethernet. Cheaper, multi-vendor (Broadcom, Arista, Cisco, Nvidia's own Spectrum-X) and familiar to every data-centre team, but classic Ethernet drops packets under congestion and needs careful tuning (priority flow control, congestion control) to behave.
Ultra Ethernet
The Ultra Ethernet Consortium (founded 2023) published its 1.0 specification in June 2025 (now v1.0.3, July 2026). It adds packet spraying across many paths, a new transport and modern congestion control, and does not need a lossless fabric. Compliant NICs and switches ramp through 2026โ€“2027.

Real deployments show both work at enormous scale. Meta built two 24,576-GPU clusters in 2024 with identical GPUs and 400 Gb/s endpoints, one on RoCE (Arista 7800 switches) and one on InfiniBand, and trained Llama 3 on both. xAI's 100,000-GPU Colossus in Memphis runs on Nvidia's Spectrum-X Ethernet; Nvidia says it kept 95% data throughput against about 60% for standard Ethernet (a vendor figure).

โš ๏ธ Has Ethernet already won? Analysts agree the trend: InfiniBand held roughly 80% of AI back-end networks in late 2023, and Ethernet has gained fast since. They disagree on timing: one summary of Dell'Oro data says Ethernet already passed InfiniBand and reached about two-thirds of the market by early 2026; another says the crossover is forecast for 2027. Either way, Nvidia sells both, and Spectrum-X is one of its fastest-growing products.

๐Ÿ’ฅ Why AI traffic collides

Ordinary data-centre traffic is millions of small, random flows, and classic Ethernet load-balancing (ECMP) hashes each flow onto one path. That works when flows are many and small. AI training is the opposite: a few elephant flows, all starting at the same instant, all needed before anyone can continue. If the hash happens to put two elephants on one link, that link runs at half speed and every GPU waits for the slowest transfer. Packet spraying (in Spectrum-X, Ultra Ethernet and InfiniBand adaptive routing) splits every flow across all paths instead.

๐ŸŽฒ Simulator: flow hashing vs packet spraying

8 large flows cross 8 equal paths between two switches. With per-flow hashing, each flow lands on a random path. Press "New collective" to start another training step.

Teaching model: each flow carries one unit of data, each path carries one unit per time step, so a step lasts as long as the busiest path. Real fabrics also face incast at the receiver, retransmissions and stragglers.

๐Ÿชž Google's other path: optical circuit switches

Google took a different route for its TPUs. Instead of a packet switch that reads every packet, its Apollo optical circuit switches use arrays of tiny MEMS mirrors to steer light from one fibre straight into another. They do not look at packets at all; they just set up a path, like an old telephone exchange. TPU v4 used them to wire 4,096 chips into a pod whose 3D-torus shape can be changed in software: cubes of 64 chips are joined in different arrangements, faulty cubes are swapped out, and a job gets the topology that suits it. Later generations (Ironwood, TPU 8) scale the same idea to pods of 9,000+ chips, with copper inside a rack and optics between racks.

๐Ÿ”€ Rewire a pod with mirrors

Four 64-chip cubes connect through an optical circuit switch. Pick a layout: the mirrors re-point and the same hardware becomes a different machine.

Simplified from Google's TPU v4 paper (Jouppi et al., ISCA 2023) and "Jupiter Evolving" (Poutievski et al., SIGCOMM 2022). TrendForce estimates an OCS uses ~100 W against ~3 kW for an electrical switch of similar role; treat as an estimate.

The trade-off: circuits change in milliseconds to seconds, not per packet, so OCS suits traffic patterns known in advance (like a training job's collectives) rather than random traffic. It saves power and money, and it made Google one of the few players never dependent on Nvidia's networking.

๐Ÿ’ก Light comes to the chip: co-packaged optics

Every scale-out link today ends in a pluggable transceiver: a module the size of a thumb drive that turns electrical signals into light. A large cluster needs several per GPU, each drawing well over 10 watts, and the electrical path from switch chip to module burns power too. At 100,000+ GPUs, optics become a meaningful share of the data centre's power and a common cause of failures. Three answers are competing:

๐Ÿ”‹ Relative optics power per bit (vendor claims)

Index: conventional DSP pluggable = 100. Lower is better.

Nvidia claims its silicon-photonics switches are about 3.5ร— more power-efficient than pluggables; LPO vendors claim roughly half the power of DSP modules. Real savings depend on reach and design.

Pluggables (DSP)
Today's workhorse, shipped in huge volumes by Coherent, Lumentum, InnoLight and others. Easy to replace when one fails.
Linear-drive (LPO)
Removes the power-hungry signal processor from the module and lets the switch chip drive the optics directly. Cheaper and cooler, but harder to make work at the highest speeds and longer reaches.
Co-packaged optics (CPO)
Puts the optical engines on the same package as the switch chip, so signals travel millimetres rather than centimetres over copper. Broadcom has shipped CPO switches since 2024; its Tomahawk 6 "Davisson" reaches 102.4 Tb/s with sixteen 6.4 Tb/s engines. Nvidia's Quantum-X Photonics InfiniBand switch (115 Tb/s, 144 ports of 800 Gb/s) shipped first, and Dell'Oro reports Spectrum-X Photonics Ethernet switches moving into production in 2026.
โš ๏ธ Early days SemiAnalysis estimates only 10,000โ€“15,000 CPO units ship in 2026. The worries are serviceability (a failed laser can mean replacing an expensive switch, which is why designs move lasers into separate replaceable modules) and supply-chain maturity. Pluggables will dominate volumes for years; CPO matters most for the biggest switches and, later, for optical scale-up.

๐ŸŒ Scale-across: one training job, many buildings

A single site can only get so much power (see the power bottleneck). So labs increasingly spread training across several data centres tens or hundreds of kilometres apart. That brings new problems: light in fibre takes about 5 microseconds per kilometre, so a 100 km link adds about a millisecond each way, and long-haul capacity is far more expensive than in-building cables.

Nvidia's answer, Spectrum-XGS (announced at Hot Chips in August 2025), adds distance-aware congestion control and telemetry; Nvidia claims nearly 1.9ร— better NCCL collective performance across sites. Arista estimates its own scale-across revenue at about $1.2 billion in 2026. On the software side, training methods that communicate less often (hierarchical or asynchronous gradient exchange) reduce how much traffic must cross the long links.

๐Ÿ•ฐ๏ธ Timeline: how the wars started

2016 ยท NVLink 1 debuts on the P100 GPU, offering a faster path than PCIe.
2018 ยท NVSwitch arrives in the DGX-2, making 16 GPUs fully connected.
2019โ€“2020 ยท Nvidia agrees to buy Mellanox (about $6.9B), bringing InfiniBand and Ethernet NICs and switches in-house.
2022โ€“2023 ยท Google publishes its optical circuit switching (Apollo) and TPU v4 papers.
Jul 2023 ยท Ultra Ethernet Consortium launched by AMD, Arista, Broadcom, Cisco, Meta, Microsoft and others.
Mar 2024 ยท GB200 NVL72 announced: 72 GPUs in one NVLink domain over a copper spine. Meta details its twin 24K-GPU RoCE and InfiniBand clusters.
May 2024 ยท UALink promoter group formed (AMD, Google, Intel, Meta, Microsoft, Broadcom, Cisco, HPE).
2024 ยท xAI's 100,000-GPU Colossus goes live on Spectrum-X Ethernet.
Apr 2025 ยท UALink 1.0 specification: up to 1,024 accelerators in one switching level.
May 2025 ยท Nvidia opens NVLink to other chips with NVLink Fusion.
Jun 2025 ยท Ultra Ethernet 1.0 specification published.
Aug 2025 ยท Spectrum-XGS "scale-across" Ethernet announced.
Early 2026 ยท ESUN 1.0 (Ethernet for scale-up) approved at OCP; Broadcom backs Ethernet scale-up over UALink.
Mar 2026 ยท Nvidia invests $2B in Marvell, which will build NVLink Fusion-compatible scale-up parts.
Apr 2026 ยท UALink 2.0 specification.
2026 ยท Vera Rubin NVL72 with NVLink 6 (3.6 TB/s per GPU). CPO switches enter production. Cornelis announces a UALink/ESUN fabric (Sep).

Purple = open-standard milestones; teal = Nvidia and hyperscaler milestones.

๐Ÿ“‹ The scorecard

Interconnects side by side

Simplified. "Status" as of October 2026.

TechnologyTierBackersScaleStatus
NVLink + NVSwitchScale-upNvidia (Fusion opens it to partners such as Marvell)72 GPUs per rack today; 3.6 TB/s per GPU (NVLink 6)Shipping in volume
UALinkScale-upAMD, Intel, hyperscalers, Cornelis and othersUp to 1,024 accelerators in one switch levelSpec 1.0 (2025) and 2.0 (2026); products 2026โ€“2027
ESUN / SUE-TScale-upMeta, Microsoft, Broadcom, Arista (OCP)Ethernet switch scale; slim 4-byte headerESUN 1.0 approved 2026; early products
Google ICI + OCSScale-up / podGoogle (TPU only)Thousands of TPUs per reconfigurable podIn production since TPU v4
InfiniBandScale-outNvidiaTens of thousands of GPUs; 800 Gb/s portsMature; losing share to Ethernet
Ethernet (RoCE / Spectrum-X)Scale-outNvidia, Broadcom, Arista, Cisco100,000+ GPUs (Colossus)Shipping; growing fastest
Ultra EthernetScale-outUEC members (AMD, Broadcom, Cisco, Meta, Microsoftโ€ฆ)Designed for millions of endpointsSpec 1.0.3; silicon ramping 2026โ€“2027
Spectrum-XGS etc.Scale-acrossNvidia; Arista and others with own productsMultiple sites, up to hundreds of kmEarly deployments (vendor-reported)

๐Ÿ”ญ What to watch in 2027

๐ŸŽฏ The bottom line The AI "computer" is no longer a chip but a building, and the network is what makes it one machine. Nvidia leads at every layer: NVLink inside the rack, InfiniBand and Spectrum-X across the hall, photonics and scale-across beyond. The rest of the industry is betting that open standards (UALink, ESUN, Ultra Ethernet) will do to Nvidia's networking what Ethernet did to every proprietary network before it. Scale-out is already tilting toward Ethernet; scale-up, the hardest and most valuable layer, is where the war will be decided.

๐Ÿ”ข Reading the numbers

๐Ÿ“š References and papers