Making 100,000 GPUs Talk: The Engineers Who Run an AI Factory's Network
A training job is only as fast as its slowest message. Every step, thousands of GPUs swap gradients in synchronised bursts, and a single congested link or badly placed flow slows them all. The hardware side of that story is in Interconnect Wars. This part is about the software and the people: the fabric engineers who configure RDMA, congestion control and routing, and the collective-communication engineers who write the libraries that decide how GPUs talk. Meta's published account of its RoCE network is the main case study.
1. Where we are in the stack
The AI factory stack, by role
2. The roles: fabric engineer and collectives engineer
Two kinds of engineer share this layer. The network or fabric engineer designs, configures and operates the physical-logical network. Sciforium's posting for a GPU Cluster Engineer, Networking asks for 7+ years running production data-centre networks including at least one large HPC/AI fabric (InfiniBand or RoCEv2), hands-on tuning of lossless RoCEv2 (PFC, ECN, DCQCN) and/or InfiniBand fabric management, across everything from the RDMA compute fabric to cross-site links. MBZUAI's Institute of Foundation Models lists InfiniBand, RoCE, GPUDirect RDMA and integrating RDMA into Kubernetes workloads; EPAM's principal HPC network role asks for 4+ years on GPU cluster networking within 8+ years overall.
The collective-communication engineer works in software: libraries such as NVIDIA's NCCL, AMD's RCCL or Meta's NCCLX that implement all-reduce, all-gather and friends, and that increasingly take on jobs the network used to do, such as managing congestion.
Two roles, two skill profiles
Our reading of the postings and papers (1โ5). The day timeline is a composite illustration.
3. What lives at this layer
The network software stack
4. Why AI traffic is hard
Ordinary data-centre traffic is millions of small, independent flows; hashing each flow onto one of several equal-cost paths (ECMP) spreads them well. Meta's paper describes AI training traffic as the opposite: far fewer flows with little diversity, bursty on-off behaviour, and "elephant" flows that run near line rate. With so few large flows, random hashing regularly puts two of them on the same link while another link sits idle, and because the job is synchronised, the whole step waits for the slowest link.
ECMP collision simulator
Illustrative. Large flows are hashed onto equal-cost uplinks. When two land on one link, each gets half the bandwidth, and the synchronised step waits for them.
5. Case study: Meta's RoCE network
Meta chose RoCEv2 (RDMA over Converged Ethernet) as the inter-node transport for most of its AI capacity, and its 24,000-GPU RoCE cluster trained a large Llama 3 variant on 16,000 GPUs. The SIGCOMM 2024 paper is unusually candid about how the design evolved:
Who controls congestion?
Illustrative comparison of the two approaches Meta describes.
6. The collective library: NCCLX
Collectives are the communication patterns of distributed training: all-reduce to sum gradients, all-gather and reduce-scatter for sharded models (see Producing Intelligence Part 4). Meta's 2025 paper "Collective Communication for 100k+ GPUs" describes NCCLX, its framework for collectives on clusters of more than 100,000 GPUs for both training and inference. Its custom transport, CTran, is host-driven, zero-copy and "SM-free": it moves data without occupying the GPU's compute units, leaving them for the model. Meta reports substantial communication-efficiency gains on Llama 4, and the code is public in its torchcomms repository.
Ring vs tree all-reduce
Illustrative cost model. Rings use bandwidth efficiently for large messages; trees take fewer latency-bound steps for small ones. Libraries pick per message size.
7. Standards and what's next
Ethernet vendors are standardising what hyperscalers built in-house. The Ultra Ethernet Consortium published its 1.0 specification on 11 June 2025, about a year later than planned. It spans NICs, switches, optics and cables and defines RDMA over Ethernet and IP, congestion control and a new transport, Ultra Ethernet Transport (UET), with libfabric as the programming interface; HPE says about 75% of UET draws on its Slingshot transport (a vendor claim). NVIDIA's Spectrum-X Ethernet takes a vertically integrated route with adaptive routing and congestion control across its switches and NICs, and InfiniBand remains common in many clusters. Hardware comparisons are in Interconnect Wars.
8. When it breaks
Network detective
Pick a symptom. Simplified from operator practice; real triage uses switch counters, NIC telemetry and nccl-tests.
9. Handoffs and careers
Who the network team works with
- Typical backgrounds
- Data-centre and HPC network engineers, telecom engineers, networking researchers; for collectives, HPC/MPI and systems programmers.
- Core skills
- RDMA (verbs, RoCE, InfiniBand), PFC/ECN/DCQCN, Clos design, ECMP and adaptive routing, switch OS and automation; for collectives, C++/CUDA, NCCL internals and performance analysis.
- Growth path
- Network engineer โ fabric architect for a site or region; collectives engineer โ owner of the communication library or ML systems lead.
- Further reading
- Interconnect Wars ยท Splitting a Giant ยท Debugging a Distributed Job
๐๏ธ Case file: Meta
- Transport
- RoCEv2 for most AI capacity; 24K-GPU RoCE cluster trained a Llama 3 variant on 16K GPUs.
- Topology
- Separate backend network; star โ two-stage Clos "AI Zone".
- Routing
- ECMP โ path pinning โ ๏ธ secondary โ centralised traffic engineering + enhanced ECMP.
- Congestion
- DCQCN โ collective-library-managed, receiver-driven admission, co-tuned with the network.
- Library
- NCCLX for 100K+ GPUs; CTran host-driven, zero-copy, SM-free transport; open in torchcomms.
๐ Next in the series
Part 4: Storage and data movement โ the engineers who keep petabytes of training data and multi-terabyte checkpoints flowing to and from tens of thousands of GPUs.
๐ข Reading the numbers
- Meta's design evolution (RoCEv2, backend network, AI Zone, traffic characteristics, traffic engineering + enhanced ECMP, DCQCN โ collective-library congestion management, receiver-driven admission) comes from the SIGCOMM 2024 paper and Meta's engineering post as quoted in search results; path pinning comes from a secondary summary โ ๏ธ.
- NCCLX facts come from arXiv 2510.20171; we do not quote specific Llama 4 speed-up figures because we could not verify them.
- Ultra Ethernet 1.0 comes from consortium coverage; the 75% Slingshot figure is HPE's claim.
- Role descriptions come from Sciforium, MBZUAI/IFM and EPAM postings.
- The collision simulator, all-reduce model, congestion diagram and detective picker are illustrations.
๐ References
- Meta
- Gangidi et al.: "RDMA over Ethernet for Distributed AI Training at Meta Scale" (SIGCOMM 2024)
- Meta Engineering: RoCE networks for distributed AI training at scale
- Meta: "Collective Communication for 100k+ GPUs" (NCCLX, 2025)
- Talent500: summary of Meta's RoCE paper (secondary)
- Standards
- Linux Foundation: UEC launches Specification 1.0 ยท HPE on UEC 1.0
- Roles
- Sciforium: GPU Cluster Engineer, Networking (posting)
- MBZUAI/IFM: Senior Network Engineer, Supercomputing (posting)
- EPAM: Principal HPC Network Engineer (posting)
- Related on this site
- Interconnect Wars ยท Part 2: OS, drivers & runtime ยท Producing Intelligence Part 1