ThisCalculator

Everyday

Ethernet Scale-Up Networking in AI Clusters: How It Works

Updated · Standard formula, shown below

This guide explains how Ethernet-based scale-up networking operates in AI clusters, a design that lets hundreds of GPUs communicate as if they were one giant device. You’ll learn the core concepts—fat-tree topologies, RoCE, and congestion control—and the practical steps to plan, configure, and troubleshoot such a network.

What You’ll Need

  • Basic understanding of TCP/IP and Ethernet switching
  • Familiarity with AI training workloads (e.g., data parallelism, model parallelism)
  • Access to network switch configuration interfaces (CLI or web)
  • Knowledge of your GPU server’s NIC specs (25/100/200/400GbE)
  • Optional: a network simulator for testing topologies

Step-by-Step

  1. Understand the goal: Scale-up networking connects GPUs within a single training job to minimize latency and maximize bandwidth. Unlike scale-out (connecting many jobs), scale-up needs extremely high throughput and low jitter.
  2. Choose a topology: The most common is a fat-tree (Clos) topology. For a cluster with N GPUs, use leaf switches (each connecting a rack of servers) and spine switches (interconnecting leaves). A two-tier fat-tree with k ports per switch supports k^2/4 GPUs. For larger clusters, add a third tier.
  3. Select the right Ethernet variant: Standard Ethernet has high overhead and lossy behavior, which hurts AI training. Use RoCEv2 (RDMA over Converged Ethernet v2) to provide RDMA semantics—direct memory-to-memory transfers without CPU involvement. Alternatively, consider InfiniBand but note that Ethernet with RoCEv2 is cost-effective and widely supported.
  4. Enable lossless or loss-tolerant transport: RoCEv2 requires a lossless network to perform well. Configure Priority Flow Control (PFC) on all switch ports to prevent packet drops for RoCE traffic. Set up Explicit Congestion Notification (ECN) for congestion marking and Data Center Quantized Congestion Notification (DCQCN) for end-to-end flow control.
  5. Design the IP addressing scheme: Assign each GPU a unique IP address. Use a private /16 or /8 subnet. For example, 10.0.0.0/16 gives you 65,534 addresses, enough for most clusters. Keep the subnet flat to avoid routing complexity.
  6. Configure switches: On each leaf and spine switch:
    • Enable PFC on the RoCE priority queue (e.g., priority 3).
    • Set up ECN marking thresholds (e.g., mark when queue depth exceeds 100KB).
    • Disable flow control for non-RoCE traffic.
    • Use static routing or OSPF for spine-leaf connectivity—simpler than BGP for small clusters.
  7. Configure GPU servers: On each server, install the NVIDIA Mellanox driver (if using NVIDIA NICs) or the appropriate vendor driver. Enable RoCEv2 on the NICs, set the MTU to 9000 (jumbo frames), and assign IP addresses to each NIC.
  8. Tune the host stack: Set the receive queue size, enable adaptive coalescing, and adjust the interrupt moderation to reduce latency. Use ethtool to set the flow control and offload features (e.g., tx-checksum-ipv4 on).
  9. Test connectivity: Use ib_write_bw (if using InfiniBand verbs) or perftest to measure bandwidth and latency between two GPUs. Expect near-line-rate bandwidth (e.g., 95% of 100GbE) and latency under 2 microseconds for a single switch hop.
  10. Scale up the test: Run a multi-node test using NCCL (NVIDIA Collective Communications Library) to simulate all-reduce operations. Use nccl-tests to measure bus bandwidth and latency. If performance is lower than expected, check for congestion or packet drops.
  11. Monitor and tune: Use switch counters to monitor PFC pause frames and ECN marks. If pause frames are frequent, your network is oversubscribed—add more spine links or reduce the number of servers per leaf. Also, consider using adaptive routing (if your switches support it) to balance traffic.

Common Mistakes

  • Mixing RoCE and normal traffic on the same priority: This causes head-of-line blocking. Always separate RoCE traffic into its own priority class.
  • Oversubscribing the spine links: A common design error is having fewer spine links than leaf links. For scale-up, you need 1:1 oversubscription (equal bandwidth up and down).
  • Ignoring PFC deadlocks: If PFC is misconfigured, it can cause a deadlock where switches wait for each other. Use PFC only on the RoCE queue and set a reasonable buffer size.
  • Using default MTU: The default 1500 MTU adds overhead. Use 9000-byte jumbo frames to reduce packet count and CPU usage.
  • Not enabling ECN: Without ECN, congestion leads to packet drops, which severely degrade RoCE performance. Always enable ECN with DCQCN.

Tips

  • Start with a small test cluster (e.g., 8 GPUs) to validate your configuration before scaling to hundreds.
  • Use the same switch vendor across the entire fabric to ensure consistent QoS behavior.
  • Consider using lossless Ethernet with PFC, but also test loss-tolerant modes if your workload can handle occasional drops.
  • Use network simulation tools (e.g., OMNeT++ with INET) to model your topology before deployment.
  • Document your QoS configuration—it’s easy to forget which queue is RoCE.

Troubleshooting

  • Low bandwidth: Check for packet drops (ethtool -S eth0 | grep drop). If drops are present, enable PFC or increase buffer sizes. Also verify that both ends use the same MTU and flow control settings.
  • High latency: Measure latency with ping -M do -s 8972 (for jumbo frames). If latency is above 10 microseconds, check for switch congestion or incorrect interrupt coalescing.
  • PFC pause frames storm: This indicates that a receiver is overwhelmed. Check the receiving NIC’s buffer size and reduce the PFC threshold. Also, ensure that the sending switch is not oversubscribed.
  • NCCL errors: If NCCL reports timeout, check the IP routing and that all GPUs can ping each other. Also, verify that the NCCL library is built with RoCE support (e.g., NCCL_PROTO=Simple).
  • Inconsistent performance: Run a second test to see if the issue is persistent. If it varies, check for background traffic or a failing cable. Use cable test on the switch.

Frequently Asked Questions

Why use Ethernet instead of InfiniBand for AI scale-up?

Ethernet is cheaper, more familiar, and easier to integrate with existing data centers. With RoCEv2, it can achieve similar performance to InfiniBand for many AI workloads, especially with modern 400GbE switches.

What is the difference between scale-up and scale-out networking?

Scale-up connects GPUs within a single job to act as one large GPU, requiring very high bandwidth and low latency. Scale-out connects many jobs or users, prioritizing throughput and isolation.

Do I need special switches for RoCE?

Yes, you need switches that support PFC, ECN, and DCQCN. Most modern data center switches (e.g., from Cisco, Arista, NVIDIA) support these features, but you must enable them.

How many GPUs can I connect with a two-tier fat-tree?

With a two-tier fat-tree using 64-port switches, you can connect up to 1,024 GPUs (64 leaves × 16 servers each, with 32 spine switches). For larger clusters, add a third tier.

What is the typical latency for RoCEv2 in a fat-tree?

Within a single leaf switch, latency is around 1-2 microseconds. Across spine switches, it’s 2-5 microseconds, depending on the number of hops and switch cut-through latency.

Can I use standard Ethernet NICs for RoCE?

Yes, but you need NICs that support RoCEv2 hardware offload. Most modern NICs from Mellanox (now NVIDIA), Intel, and Broadcom support it. Check the spec sheet for RoCE support.

Related

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.