This is part 3 of my series documenting my journey learning about neural network (AI) infrastructure. In part two, I went through AI training workflows and which parts of the workflow hit the network.

In this post, I go through the routing side of AI networks, with a focus on Equal-Cost Multi-Path (ECMP) and why AI network engineers care so much about ECMP.

L3 CLOS Topologies - Frontend + Backend

In part two, I described the split between the frontend and backend networks. Tom Emmons from Arista in his 2025 presentation Arista Networking for AI: The Ethernet Backplane said something interesting about frontend networks. He said the frontend “looks like a traditional data center. You just care about it a lot more, because you have billions of dollars of hardware behind it instead of millions.” With the rise of inference, I believe frontend networks are going to start to matter just as much as backend networks. So it seems like RAG, KV caches and prefill may play a greater role in the AI networking design? I will update this blog post as I learn more about this.

What is consistent, though, between the frontend and backend networks is the foundation of an AI network: the L3 CLOS topology (leaf-spine). I suspect it’s the cost that has governed this decision. Ethernet/IP is a well-known technology, and you can find engineers to troubleshoot and manage it. L3 CLOS deployments can also be easily automated, because both servers and switches are programmable using infrastructure-as-code tooling.

Why L3 and not one big Layer 2 network? The best reference I found is RFC 7938 - Use of BGP for Routing in Large-Scale Data Centers, written in 2016 by Petr Lapukhov et al. The authors stated that, based on their experience, the operational simplicity and network stability are achieved using L3 CLOS with a small number of staff.

IPv4 or IPv6?

During Emmons’ talk, someone in the audience asked: “Is IPv6 required for back-end networks?” I thought this was a great question.

I think based on Emmons’ answer, it makes sense, at least to me, to use IPv6 for address simplicity, especially now that you can use infrastructure-as-code and easily create non-overlapping addresses.

He also said you won’t see IPv6 in scale-up networks. My guess is that any additional header info increases latency.

Scale Up - Ethernet’s Next Frontier?

In part two, I described the scale-up network as NVLink, connecting GPUs inside a server or rack. Emmons said Arista believes Ethernet is a strong competitor for the scale-up fabric. His reasons were that Ethernet is cost-effective and, from a supply-chain perspective, there are many vendors to pick from.

But scale-up is hard for Ethernet.

  • Bandwidth: A GPU might have 400G of NIC bandwidth for scale-out, but 3.6 Tb/s of scale-up bandwidth. That’s 8 to 9 times more.
  • Cost and power: Building a power-efficient Ethernet scale-up network is a current challenge.
  • Small packets: GPUs talk to each other in the scale-up network using memory load and store operations, which produce 256-byte, cache-line-sized packets. Traditional Ethernet is not good at managing this packet size.

But Arista continues to venture into scale-up networking. Arista announced on October 7, 2026 that it supports ESUN (Ethernet for Scale-Up Networking). Perhaps Arista will pitch Ethernet as the cost-effective way to build scale-up networks for AI inference? I believe AI Inteference doesn’t need the high bandwidth that AI training does, so scale-up Ethernet for AI inference may just work?

Network Depth - Why AI Fabrics Prefer 2 Tiers

Back to the scale-out network. How deep should a CLOS be? I kept hearing “2-tier” and “3-tier” in these videos, so here are the terms as I understand them:

  • Single tier: one big modular switch. Every GPU connects to the same chassis.
  • 2-tier: leaf-spine. A packet goes leaf, spine, leaf. This is a 3-stage CLOS network.
  • 3-tier: leaf, spine, super spine. This is a 5-stage CLOS network.

From the videos, AI network architects prefer 2 tiers whenever possible. The reasoning is:

  • Fewer tiers are cheaper, more performant and use less power.
  • It’s a cost issue. Every tier adds two more optics per GPU. Every GPU has a 400G or 800G port, so that’s a lot of optics.
  • Every tier makes load balancing harder.
  • Every tier adds more congestion points for traffic to run into. AI scale-out networks, from what I’ve heard so far, still struggle with congestion management.

A 2-tier switching topology seems to be a good compromise and appears to be the most prevalent scale-out topology.

Meta’s SIGCOMM 2024 talk by Adithya Gangidi shows the same thinking in practice. Meta’s “AI Zone” is a 2-tier CLOS of 4,000 GPUs with full bisection bandwidth.

ECMP - A History and Current Challenges

Equal-Cost Multi-Path (ECMP) routing has been around for a long time. RFC 2991 and RFC 2992 described multipath forwarding and the hash-based method back in 2000. In a CLOS network, every leaf has multiple equal-cost paths to every other leaf, one through each spine. ECMP is what spreads the traffic across them.

Hugh Holbrook of Arista explained how it works in his 2024 NANOG 92 keynote. The switch feeds the 5-tuple (source IP, destination IP, protocol, source port and destination port) into a hash function in hardware. The result maps to one of the ECMP paths. Every packet in a flow has the same 5-tuple, so every packet in the flow takes the same path and arrives in order.

Why AI training breaks ECMP

Holbrook walked through a simple example that shows how a small number of high-bandwidth flows can overload a link. With only 8 flows per server, one uplink ended up carrying 140% of its capacity, and every flow on that link slowed down. With normal TCP traffic, that’s not a big deal. There are thousands of flows, they average out across the links, and no application waits on the slowest one. In the AI training universe, it’s a big deal. Collectives run at the speed of the slowest link, so one overloaded link slows down every GPU in the job, on every iteration. With a training run of 1,000+ GPUs, that’s a lot of wasted time and power. And if a link actually fails, the job has to roll back to its last checkpoint, which can cost hundreds or thousands of hours of GPU time.

And AI training has few flows. In the SIGCOMM talk, Gangidi said that out of the box, Meta saw fewer than 10 flows per collective operation, and with that few flows, hash-based ECMP “will perform very poorly.” On the Packet Pushers podcast about MRC, a Broadcom engineer summed it up: “tail latency is a killer for RoCE”.

How the industry is fixing ECMP

Here’s how I understand the progression of various solutions regarding ECMP.

1. Make more flows

Creating more flows is one solution AI network engineers tried. Meta took the messages, or “chunks” as the NCCL paper I referenced in part two calls them, and found a way to send each chunk on a different queue pair, round-robin. Based on the RoCEv2 specification, I believe each connected queue pair keeps a fixed UDP source port, and different queue pairs usually get different ports. More queue pairs mean more UDP source ports. That creates higher entropy that switches can use when hashing traffic across the 2-tier network.

2. Dynamic Load Balancing (DLB)

The next step is to let the switch chip look at how busy each link is. Broadcom calls this Dynamic Load Balancing. It is a feature of the Tomahawk 5 chip, which was launched in 2022.

Here is how I understand it. Plain ECMP only looks at packet headers. DLB also tracks how much traffic each uplink has recently sent and how deep its queue is, and picks the least busy link. Most DLB deployments work on flowlets. A flowlet is a burst of packets in a flow followed by a short pause. If the pause is longer than the delay difference between the paths, the next burst can safely move to a different link without arriving out of order. AI traffic has natural pauses between compute and communication, so flowlets fit well.

Emmons said DLB is the most common solution deployed today when using Arista switches. He called it “a fundamentally greedy algorithm.” It does a great job balancing traffic leaving the leaf, but it can’t fix imbalance from the spine down to the destination leaf.

I’m still looking for good documentation on how SONiC and FBOSS expose DLB. I’ll update this post when I find it.

3. Global load balancing

DLB only sees its own links. The next step is to share congestion information between switches. Broadcom’s Tomahawk 5 Global Load Balancing uses “distributed inter-switch communication of congestion information to choose the best global path.” Tomahawk 6 extends this as Cognitive Routing 2.0.

4. Packet spraying (MRC)

Packet spraying sends the packets of a single flow across all the paths, instead of pinning the whole flow to one path.

The catch is that packets arrive out of order, so the receiving NIC has to handle it. Holbrook showed that spraying fixes his example: the worst link drops to about 90% and efficiency goes to 99.98%. Emmons said packet spraying is where the industry is heading, and multiple customers have deployed it. “It’s just a matter of the NICs getting there.”

This packet spraying workflow is the basis for Multipath Reliable Connection (MRC). In the Packet Pushers episode, three Broadcom engineers explained that MRC is an open extension to RoCEv2, driven by Microsoft and OpenAI with AMD, Intel, NVIDIA and Broadcom. The NIC puts a different entropy value on each packet, and unmodified ECMP switches spray them across every path. It is not clear to me yet if this is implemented today in any NIC.

5. Ultra Ethernet

The UEC approach to fixing ECMP problems is similar to MRC: spray the packets across all the paths. When a switch queue is full, instead of dropping a packet, the switch trims it down to 64 bytes, changes the DSCP value and sends it to the receiver on a high-priority queue. This way the receiver knows right away what was lost. UEC also adds a mechanism where the receiver hands out credits that tell each sender how much data it can send. Surprisingly, UEC doesn’t require new switches. Holbrook said a switch can be “completely compliant” by supporting Ethernet as it is today. Trimming is optional and needs switch support.

Both solutions require NICs that support their protocol. For either UEC’s ECMP changes or OCP’s MRC, packet trimming needs switch hardware support. Broadcom lists packet trimming as a feature of the Tomahawk 6. I’m not sure if that’s already in place in SONiC or FBOSS.

References: