ThisCalculator

Everyday

AI Data Center Hardware: What You Need Besides GPUs

Updated · Standard formula, shown below

This article explains what hardware an AI data center needs besides GPUs, covering the essential infrastructure—CPUs, memory, storage, networking, power, and cooling—that makes GPU clusters actually work. You’ll get a clear breakdown of each component’s role, how they fit together, and what to consider when planning or upgrading an AI facility.

Why GPUs Aren’t Enough

Graphics processing units (GPUs) do the heavy lifting for AI training and inference, but they are useless in isolation. A GPU server is like a high-performance engine: it needs a fuel system, a cooling system, a chassis, and a driver to move. In a data center, those supporting roles are filled by central processing units (CPUs), high-speed memory, fast storage, low-latency networking, and a robust power and cooling infrastructure. Without these, GPUs would sit idle, starved of data or overheating, and the entire cluster would underperform or fail.

Central Processing Units (CPUs) and Motherboards

CPUs manage the overall system: they run the operating system, orchestrate data flow, handle pre-processing and post-processing of data, and coordinate communication between GPUs and other components. In AI workloads, CPUs are often the bottleneck if they are too slow, so choosing the right CPU matters.

  • Core count and clock speed: AI servers typically use server-class CPUs with many cores (e.g., 32 to 64 cores) and high memory bandwidth. For example, AMD EPYC and Intel Xeon Scalable processors are common.
  • PCIe lanes: Each GPU needs a PCIe connection (typically x16 lanes). A motherboard must have enough lanes to connect all GPUs without reducing bandwidth. For 8 GPUs, that means at least 128 PCIe lanes from the CPU(s).
  • Memory channels: CPUs with more memory channels (e.g., 8 or 12 channels) allow faster access to RAM, which is critical when feeding data to GPUs.

Motherboards for AI servers are specialized—they have multiple GPU slots (often 4 or 8), redundant power connectors, and robust VRM (voltage regulator module) designs to handle the power draw of multiple GPUs.

Memory: RAM and High-Bandwidth Memory

AI models and datasets are large, and data must be quickly accessible to GPUs. Two types of memory are involved:

  • System RAM (DDR4/DDR5): This is the main memory for the CPU. It stores the operating system, frameworks, and data batches before they are sent to GPUs. For a server with 8 GPUs, 512 GB to 1 TB of RAM is common.
  • GPU memory (HBM, GDDR6): This is the memory physically on the GPU card. It holds the model weights and intermediate tensors during training. For large models, GPUs with 40 GB to 80 GB of memory (like NVIDIA A100 or H100) are used, and sometimes multiple GPUs are needed to fit a model.

The speed of memory matters. High-bandwidth memory (HBM) on GPUs is extremely fast, but system RAM speed also affects data loading. Using slower RAM can cause the CPU to take longer to prepare data, leaving GPUs idle.

Storage: Fast and Scalable

AI training requires massive amounts of data—often terabytes or petabytes. Storage must be fast enough to feed data to GPUs without delays, and scalable enough to hold all the data.

  • NVMe SSDs: For the hot tier (data being actively used), NVMe drives over PCIe offer very high throughput (several GB/s). They are used for the training dataset and checkpointing.
  • Parallel file systems: Systems like Lustre or GPFS (IBM) are designed to handle concurrent reads from many servers. They stripe data across many drives to maximize bandwidth.
  • Object storage: For cold storage (archived datasets), object stores like Ceph or MinIO are used. They are cheaper but slower.
  • Capacity planning: A rule of thumb is to have at least 10x the size of your largest dataset in fast storage, and more in cold storage. For example, if you train on 100 TB of data, you might have 1 PB of NVMe and 5 PB of object storage.

Networking: The Backbone of Distributed Training

When training large models, data is distributed across multiple GPUs and servers. They must communicate frequently to synchronize gradients. This requires a high-speed, low-latency network.

  • InfiniBand or high-speed Ethernet: InfiniBand (e.g., HDR 200 Gb/s) is common in high-performance computing (HPC) because of its low latency and high throughput. Ethernet (100 GbE, 400 GbE) is also used, especially with RDMA (Remote Direct Memory Access) to reduce CPU overhead.
  • Network topology: A fat-tree or torus topology is often used to avoid bottlenecks. For example, a non-blocking fat-tree ensures that any server can communicate with any other at full bandwidth.
  • Switches and cables: High-port-count switches (e.g., 32 or 64 ports) and optical cables are required. The number of switches depends on the number of servers and the topology.

Without a good network, distributed training can slow down significantly. For instance, if a model requires frequent all-reduce operations, a slow network can cause GPUs to wait for data, reducing utilization.

Power and Cooling Infrastructure

AI data centers consume enormous amounts of power—a single GPU server can draw 3-5 kW, and a rack of 8 servers can draw 30-40 kW. This requires robust power distribution and cooling.

  • Power distribution: Each rack needs dedicated power circuits, often 3-phase 208V or 400V. Uninterruptible power supplies (UPS) and backup generators are essential for uptime.
  • Cooling: Air cooling is common for racks up to 20 kW, but for higher densities, liquid cooling is more efficient. Direct-to-chip cooling or immersion cooling can handle 50-100 kW per rack. For example, NVIDIA’s H100 systems often use liquid cooling in dense configurations.
  • Heat rejection: Chillers, cooling towers, or dry coolers are needed to remove heat from the building. The efficiency is measured by PUE (Power Usage Effectiveness), with modern facilities aiming for 1.1-1.3.

Power and cooling are often the largest operational costs after hardware, and can be a limiting factor in where to build a data center.

How to Choose the Right Hardware

Selecting the right components depends on your workload: training vs. inference, model size, and budget. Here are key considerations:

  1. Determine the scale: How many GPUs do you need? For a small research cluster, 4-8 GPUs might suffice. For large training runs, hundreds or thousands are needed.
  2. Match CPU to GPU: A balanced server might have 2 CPUs with 64 cores each for 8 GPUs. Use the CPU’s PCIe lanes to ensure full bandwidth.
  3. Size the memory: For training, you need enough system RAM to hold the dataset batches and model checkpoints. A common ratio is 0.5-1 GB of system RAM per 1 GB of GPU memory.
  4. Plan storage for checkpoints: Training checkpoints can be huge (multiple GBs). Ensure your storage can handle frequent writes without slowing down.
  5. Design the network: For clusters larger than 4 servers, a high-speed network with RDMA is essential. Consider the cost of switches and cables.
  6. Calculate power and cooling: Estimate the total power draw of all servers and design cooling accordingly. Liquid cooling is worth it for high-density racks.

Common Mistakes and Misconceptions

One common mistake is underestimating the CPU requirements. Some assume GPUs do all the work, but a slow CPU can starve the GPUs. Another is ignoring the network: using regular Ethernet without RDMA can cause severe bottlenecks in distributed training.

Another misconception is that more GPUs always means faster training. In reality, if the network or storage can’t keep up, adding GPUs can even slow things down due to synchronization overhead. Also, many forget about the cost of storage—high-speed NVMe is expensive, and you need a lot of it for large datasets.

Cost Considerations

The cost of AI data center hardware goes beyond the GPUs. CPUs, memory, storage, networking, and power/cooling can add 30-50% to the total hardware cost. For example, a server with 8 GPUs might cost $200,000 for GPUs alone, but the full server with CPUs, RAM, SSDs, and networking could be $300,000. Additionally, the network infrastructure for a cluster of 100 servers can cost hundreds of thousands of dollars.

Operational costs include electricity (a 1 MW facility might cost $1M per year) and cooling (which can be 30-40% of electricity). These are often overlooked in initial budgets.

Alternatives to Building Your Own

If building a data center is too expensive or complex, you can rent hardware from cloud providers (AWS, Google Cloud, Azure) or use specialized AI services. This avoids upfront costs and allows you to scale on demand. However, for long-term, heavy usage, owning hardware can be more cost-effective.

Frequently Asked Questions

Q: How much CPU do I need per GPU?
A: A common rule is to have at least one CPU core per GPU, but for data-heavy workloads, you might need 2-4 cores per GPU. For example, an 8-GPU server often has 2 CPUs with 32-64 cores total.

Q: What is the difference between InfiniBand and Ethernet for AI?
A: InfiniBand offers lower latency and higher bandwidth (up to 400 Gb/s) and is often used in HPC. Ethernet is more common and cheaper, but with RDMA and 100/200/400 GbE, it can be sufficient for many AI workloads.

Q: Do I need liquid cooling for AI servers?
A: Not always. Air cooling works for racks up to about 20 kW. For higher densities (e.g., 40 kW per rack), liquid cooling is more efficient and can reduce energy costs.

Q: How much storage do I need for AI training?
A: It depends on your dataset size. A good starting point is to have at least 10x your largest dataset in fast NVMe storage, and more in slower object storage for archives.

Q: Can I use consumer-grade hardware for AI?
A: Yes, for small experiments, but server-grade hardware is more reliable and has better support for multiple GPUs, ECC memory, and higher power delivery.

Q: What is PUE and why does it matter?
A: PUE (Power Usage Effectiveness) is the ratio of total facility power to IT equipment power. A lower PUE means more efficient cooling and power distribution. Typical AI data centers aim for 1.1-1.3.

Related

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.