Data Center Interconnect for AI: FAQ
Data center interconnect (DCI) for AI refers to the high-speed networking links and equipment that connect separate data centers so they can share compute, storage, and data for artificial intelligence workloads. This article answers the most common questions about DCI for AI, covering what it is, why it matters, how it works, and what to consider when building or upgrading one.
Background: Why AI Changes Data Center Interconnect
Traditional DCI linked data centers for backup, disaster recovery, and modest data sharing. AI workloads, especially training large language models and other deep neural networks, are different. Training a single model can require thousands of GPUs or TPUs working in parallel. When those servers are spread across more than one building or campus, the interconnect must move enormous volumes of data with very low latency and essentially zero packet loss.
Three AI-specific factors drive DCI requirements:
- Distributed training: A model may be too large for one data center, so training is split across sites. Every few seconds, the sites must synchronize gradients, requiring dedicated, high-bandwidth links.
- Data gravity: AI training datasets can be petabytes in size. Moving that data between sites over a slow link creates bottlenecks that delay experiments.
- Latency sensitivity: Some AI inference (using a trained model) needs real-time responses. If the model is hosted in a remote data center, the round-trip time of the interconnect directly affects user experience.
How DCI for AI Works
A DCI system typically includes three layers: the physical fiber, the optical transport equipment, and the network protocols that carry data.
At the physical layer, most modern DCI uses dark fiber or leased wavelengths. Dense wavelength-division multiplexing (DWDM) sends many different colors of light down a single fiber pair, each carrying a separate data stream. A single fiber pair can now carry 400 Gbps or 800 Gbps per wavelength, and modern systems support dozens of wavelengths, yielding tens of terabits per second per fiber.
The optical layer uses transponders or coherent optical modules. These convert electrical signals from switches into light, and they use digital signal processing to compensate for distortion over long distances. For AI DCI, the key parameters are:
- Capacity: Total bandwidth between sites, often 400G or 800G per wavelength.
- Reach: How far the signal can travel without regeneration. Coherent optics can go 80–120 km without amplification, and with amplifiers and forward error correction, they can span thousands of kilometers.
- Latency: The speed of light in fiber is about 200,000 km/s, so a 100 km link adds roughly 0.5 ms of one-way latency. For AI training, this is often acceptable, but for real-time inference, it may be critical.
Above the optical layer, the network uses Ethernet (100G, 400G, or 800G) or InfiniBand. InfiniBand is common inside AI clusters because of its low latency and high throughput, but it is not typically used for long-haul DCI. Instead, data centers often use IP over DWDM with RDMA over Converged Ethernet (RoCE) or TCP, depending on the workload.
Key Components of an AI DCI
An AI DCI is not just a fiber link. It includes:
- Fiber plant: Dark fiber or leased wavelengths. Dark fiber gives you full control but requires you to manage the optics.
- Optical transport: DWDM mux/demux, amplifiers, and coherent transceivers. These can be embedded in switches (pluggable optics) or in separate transport platforms.
- Switches and routers: At each site, a DCI router or switch aggregates traffic from the AI cluster and sends it to the optical layer.
- Network operating system: Software to configure, monitor, and troubleshoot the link. For AI, this includes features like priority flow control and congestion management.
- Management and telemetry: Real-time monitoring of bit error rate, optical power, and latency.
Choosing DCI for AI: What to Consider
When planning a DCI for AI, the main decisions are:
- Distance: Under 2 km, you might use direct attach cables or active optical cables. Under 80 km, standard DWDM with pluggable optics works. Beyond that, you need amplification and possibly regeneration.
- Capacity: Estimate the peak bandwidth needed. A training run might need 400 Gbps per link, but if you have multiple jobs, you may need multiple wavelengths or higher-speed optics.
- Latency budget: For synchronous training, the round-trip time between sites must be low enough that the training algorithm doesn’t stall. For example, if your training step takes 100 ms, a 10 ms interconnect delay might be acceptable, but a 100 ms delay would kill performance.
- Reliability: AI training jobs can run for weeks. A single fiber cut can waste thousands of GPU-hours. Consider diverse fiber paths and automatic protection switching.
- Cost: Dark fiber vs. leased wavelengths, equipment capex vs. opex, and power consumption of the optics all factor in.
Common Problems and Misconceptions
One common misconception is that DCI for AI is just a bigger pipe. In reality, the network must be lossless or near-lossless. When a packet drops during distributed training, the whole training step may need to be redone, causing a significant slowdown. This is why RoCEv2 with priority flow control is often used, but it requires careful tuning to avoid head-of-line blocking.
Another issue is the difference between bandwidth and latency. Adding more bandwidth does not reduce latency. If your AI job is latency-sensitive, you need to shorten the distance or use techniques like data compression or model parallelism that reduce the amount of communication.
A third problem is the “last mile” within the data center. Even if the DCI link is 800 Gbps, the servers on each end must be able to generate and consume that traffic. This means high-performance NICs, proper switch buffers, and a well-designed spine-leaf topology.
Finally, don’t forget about security. DCI links often span public rights-of-way. Encryption is essential, either at the optical layer (using MACsec or IPsec) or at the application layer.
Alternatives and Comparisons
DCI is not the only way to connect AI resources. The main alternatives are:
- Single data center: Keep everything in one building. This avoids DCI entirely but limits scale and resilience.
- Cloud provider interconnects: Use a cloud provider’s backbone to connect your on-premises AI cluster to their cloud. This is easier but less customizable and may have unpredictable performance.
- Satellite or microwave links: These are rarely used for AI due to high latency and low capacity, but they can be a backup for fiber cuts.
Compared to a single data center, DCI for AI offers scale and redundancy but adds complexity and latency. Compared to using a cloud provider, a private DCI gives you full control over quality of service and data governance.
Costs and Practical Considerations
The cost of a DCI for AI varies widely. Leasing a 100 Gbps wavelength between two cities can cost thousands of dollars per month. Dark fiber leases are often cheaper over the long term but require you to buy the optical equipment. For a 10 km link, you might spend $50,000–$200,000 on equipment, not including installation. For longer links, costs rise with amplification and regeneration.
Operational costs include power for the optics, cooling, and maintenance. A 400G coherent optic might consume 10–20 watts, which is small compared to a GPU server but adds up over many links.
When budgeting, also include the cost of failure. A redundant path can double the cost but may be worth it if a single outage costs you $100,000 in lost GPU time.
Frequently Asked Questions
What is the difference between DCI and a regular internet connection?
A regular internet connection is shared, best-effort, and has variable latency. DCI is a dedicated, point-to-point link with guaranteed bandwidth and low, predictable latency. For AI training, you need the latter because the network is part of the compute fabric.
Can I use standard Ethernet switches for DCI?
Yes, many DCI links use standard Ethernet switches with pluggable coherent optics. However, you need switches that support the specific optics (e.g., 400G ZR) and have enough buffer memory to handle congestion. For long distances, you may also need separate transport equipment.
How far can a DCI link go without regeneration?
With modern coherent optics and amplifiers, links can span 1,000–2,000 km without electrical regeneration. Beyond that, you need to regenerate the signal, which adds latency and cost. For AI, most DCI links are within 100–500 km, where performance is excellent.
Does DCI latency affect AI training?
It depends on the training algorithm. Data-parallel training with frequent synchronization is sensitive to latency. Model-parallel training with less frequent communication is less sensitive. In general, a round-trip time under 1 ms is safe for most workloads, while over 10 ms can cause noticeable slowdowns.
Is DCI for AI secure?
It can be, but you must implement security measures. The fiber itself is difficult to tap, but not impossible. Use encryption at the network layer (IPsec or MACsec) and ensure physical security of the fiber path. Many AI workloads involve sensitive data, so encryption is strongly recommended.
Can I use DCI for AI inference as well as training?
Yes, but inference has different requirements. Inference is often latency-sensitive and may not need the same massive bandwidth as training. You can use a lower-capacity DCI link for inference, but you must ensure the round-trip time meets your service-level agreement.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.