Home โ€บ Blog โ€บ Networking Software
Who Runs the AI Factory ยท Part 3 ยท Networking software

Making 100,000 GPUs Talk: The Engineers Who Run an AI Factory's Network

A training job is only as fast as its slowest message. Every step, thousands of GPUs swap gradients in synchronised bursts, and a single congested link or badly placed flow slows them all. The hardware side of that story is in Interconnect Wars. This part is about the software and the people: the fabric engineers who configure RDMA, congestion control and routing, and the collective-communication engineers who write the libraries that decide how GPUs talk. Meta's published account of its RoCE network is the main case study.

FL
FrontierAGI Team
โš ๏ธ How this was researched The Meta case study comes from its SIGCOMM 2024 paper "RDMA over Ethernet for Distributed AI Training at Meta Scale" and the accompanying engineering post, plus Meta's 2025 NCCLX paper, mostly as quoted in search results. A few details come only from secondary summaries and are marked โš ๏ธ secondary. Role descriptions come from real job postings (no Meta, OpenAI or Anthropic networking postings were found). The collision simulator, all-reduce animation and troubleshooting picker are illustrations. See Reading the numbers.

1. Where we are in the stack

The AI factory stack, by role

2. The roles: fabric engineer and collectives engineer

Two kinds of engineer share this layer. The network or fabric engineer designs, configures and operates the physical-logical network. Sciforium's posting for a GPU Cluster Engineer, Networking asks for 7+ years running production data-centre networks including at least one large HPC/AI fabric (InfiniBand or RoCEv2), hands-on tuning of lossless RoCEv2 (PFC, ECN, DCQCN) and/or InfiniBand fabric management, across everything from the RDMA compute fabric to cross-site links. MBZUAI's Institute of Foundation Models lists InfiniBand, RoCE, GPUDirect RDMA and integrating RDMA into Kubernetes workloads; EPAM's principal HPC network role asks for 4+ years on GPU cluster networking within 8+ years overall.

The collective-communication engineer works in software: libraries such as NVIDIA's NCCL, AMD's RCCL or Meta's NCCLX that implement all-reduce, all-gather and friends, and that increasingly take on jobs the network used to do, such as managing congestion.

Two roles, two skill profiles

Our reading of the postings and papers (1โ€“5). The day timeline is a composite illustration.

3. What lives at this layer

The network software stack

4. Why AI traffic is hard

Ordinary data-centre traffic is millions of small, independent flows; hashing each flow onto one of several equal-cost paths (ECMP) spreads them well. Meta's paper describes AI training traffic as the opposite: far fewer flows with little diversity, bursty on-off behaviour, and "elephant" flows that run near line rate. With so few large flows, random hashing regularly puts two of them on the same link while another link sits idle, and because the job is synchronised, the whole step waits for the slowest link.

ECMP collision simulator

Illustrative. Large flows are hashed onto equal-cost uplinks. When two land on one link, each gets half the bandwidth, and the synchronised step waits for them.

5. Case study: Meta's RoCE network

Meta chose RoCEv2 (RDMA over Converged Ethernet) as the inter-node transport for most of its AI capacity, and its 24,000-GPU RoCE cluster trained a large Llama 3 variant on 16,000 GPUs. The SIGCOMM 2024 paper is unusually candid about how the design evolved:

A separate backend network. GPU training traffic got its own "backend" fabric, apart from the ordinary front-end data-centre network. The topology moved from an early star design to a two-stage Clos for AI racks, which Meta calls an AI Zone.
Routing, round one. Plain ECMP hashing was a poor fit. Meta used path pinning early on โš ๏ธ secondary, which worked when racks were fully assigned to one job without failures but caused uneven load with partial allocations, and deployed a combination of centralised traffic engineering and an enhanced ECMP scheme.
Congestion control. Meta first used DCQCN, the standard congestion-control scheme for RoCE, then moved away from it and relied on the collective library to manage congestion instead.
Receiver-driven admission. The collective library admits traffic from the receiving side, so senders do not flood a receiver, and Meta co-tuned the collective library and the network configuration together.
๐Ÿ’ก The big lesson: the boundary moved Meta's network got simpler in some ways (less reliance on in-network congestion control) because the software that generates the traffic got smarter. In AI factories the network team and the collectives team are effectively one system; that is why the two roles in this part increasingly sit together.

Who controls congestion?

Illustrative comparison of the two approaches Meta describes.

6. The collective library: NCCLX

Collectives are the communication patterns of distributed training: all-reduce to sum gradients, all-gather and reduce-scatter for sharded models (see Producing Intelligence Part 4). Meta's 2025 paper "Collective Communication for 100k+ GPUs" describes NCCLX, its framework for collectives on clusters of more than 100,000 GPUs for both training and inference. Its custom transport, CTran, is host-driven, zero-copy and "SM-free": it moves data without occupying the GPU's compute units, leaving them for the model. Meta reports substantial communication-efficiency gains on Llama 4, and the code is public in its torchcomms repository.

Ring vs tree all-reduce

Illustrative cost model. Rings use bandwidth efficiently for large messages; trees take fewer latency-bound steps for small ones. Libraries pick per message size.

7. Standards and what's next

Ethernet vendors are standardising what hyperscalers built in-house. The Ultra Ethernet Consortium published its 1.0 specification on 11 June 2025, about a year later than planned. It spans NICs, switches, optics and cables and defines RDMA over Ethernet and IP, congestion control and a new transport, Ultra Ethernet Transport (UET), with libfabric as the programming interface; HPE says about 75% of UET draws on its Slingshot transport (a vendor claim). NVIDIA's Spectrum-X Ethernet takes a vertically integrated route with adaptive routing and congestion control across its switches and NICs, and InfiniBand remains common in many clusters. Hardware comparisons are in Interconnect Wars.

8. When it breaks

Network detective

Pick a symptom. Simplified from operator practice; real triage uses switch counters, NIC telemetry and nccl-tests.

9. Handoffs and careers

Who the network team works with

๐Ÿงญ Getting into the role
Typical backgrounds
Data-centre and HPC network engineers, telecom engineers, networking researchers; for collectives, HPC/MPI and systems programmers.
Core skills
RDMA (verbs, RoCE, InfiniBand), PFC/ECN/DCQCN, Clos design, ECMP and adaptive routing, switch OS and automation; for collectives, C++/CUDA, NCCL internals and performance analysis.
Growth path
Network engineer โ†’ fabric architect for a site or region; collectives engineer โ†’ owner of the communication library or ML systems lead.
Further reading
Interconnect Wars ยท Splitting a Giant ยท Debugging a Distributed Job

๐Ÿ—‚๏ธ Case file: Meta

Transport
RoCEv2 for most AI capacity; 24K-GPU RoCE cluster trained a Llama 3 variant on 16K GPUs.
Topology
Separate backend network; star โ†’ two-stage Clos "AI Zone".
Routing
ECMP โ†’ path pinning โš ๏ธ secondary โ†’ centralised traffic engineering + enhanced ECMP.
Congestion
DCQCN โ†’ collective-library-managed, receiver-driven admission, co-tuned with the network.
Library
NCCLX for 100K+ GPUs; CTran host-driven, zero-copy, SM-free transport; open in torchcomms.
Verdict AI networking is less about raw link speed than about fitting a few huge, synchronised flows onto the fabric without collisions or congestion. Meta's experience shows the winning move was not a cleverer switch but moving intelligence into the collective library and treating network and communication software as one system. The people who can work across both are some of the scarcest in an AI factory.

๐Ÿ‘€ Next in the series

Part 4: Storage and data movement โ€” the engineers who keep petabytes of training data and multi-terabyte checkpoints flowing to and from tens of thousands of GPUs.

๐Ÿ”ข Reading the numbers

๐Ÿ“š References