Why AI Data Centers Need So Much Memory
AI data centers need so much memory because modern AI models, especially large language models and deep learning systems, must hold their entire working set of parameters, activations, and intermediate data in fast, accessible storage to avoid crippling bottlenecks. The sheer scale of these models—often with billions or trillions of parameters—means that even a single inference request can require tens or hundreds of gigabytes of high-bandwidth memory (HBM) just to function. This article explains the technical reasons behind this memory hunger, the types of memory involved, and why simply adding more DRAM isn’t always the answer.
The Core Problem: Models Are Too Big for Fast Memory
An AI model’s parameters (weights and biases) are its learned knowledge. A model like GPT-3 has 175 billion parameters; storing just those weights in 16-bit floating point format requires about 350 GB of memory. But that’s only the starting point. During inference (generating a response), the model must also store:
- Activations: the intermediate outputs of each layer as data flows through the network
- KV cache (key-value cache): for transformer models, this stores previous tokens’ attention keys and values to avoid recomputing them
- Input/output buffers: the prompt and generated text, plus any associated embeddings
For a large model, the KV cache alone can grow to tens of gigabytes for a long conversation. All this data must be accessible at extremely high speed—otherwise the GPU or accelerator sits idle waiting for data, wasting its compute power.
Why Not Just Use Regular RAM or SSDs?
Regular DRAM (like in your laptop) is too slow for AI workloads. A GPU needs data at rates of multiple terabytes per second; standard DDR5 memory offers around 50-100 GB/s per channel, while HBM (High Bandwidth Memory) delivers 1-2 TB/s per stack. Hard drives and SSDs are even slower—even NVMe SSDs top out at a few GB/s. If a model’s weights were stored on an SSD, every inference would require waiting for data to be fetched from storage, making the model impractically slow.
Furthermore, AI accelerators have limited on-chip memory (SRAM) that is extremely fast but tiny (tens of megabytes). The model must live in off-chip memory that is close enough and fast enough to feed the compute units continuously. HBM is stacked vertically to provide both high bandwidth and reasonable capacity, but it is expensive and limited in supply.
Memory Hierarchy: The Tiered Approach
AI data centers use a hierarchy of memory types, each with different speed and cost:
- SRAM: on-chip cache, fastest, but only a few MB per chip
- HBM: high-bandwidth memory, stacked on the accelerator package, provides 1-2 TB/s bandwidth, capacities up to 144 GB per package (as of 2024)
- GDDR6/GDDR6X: used in some GPUs, higher capacity but lower bandwidth than HBM
- DDR5: system RAM, slower but cheaper and higher capacity
- NVMe SSDs: for checkpointing and offloading, but not for active computation
Because HBM is scarce and expensive, data centers often use a mix: they try to fit the model in HBM, but for larger models, they may use techniques like model parallelism (splitting the model across multiple GPUs) or offloading (moving some data to system RAM or SSD during computation). However, offloading introduces latency, so it’s only used when necessary.
Training vs. Inference: Different Memory Demands
Training a model requires even more memory than inference. During training, you need to store:
- Parameters (weights)
- Gradients (for each parameter)
- Optimizer states (e.g., momentum and variance for Adam optimizer)
- Activations for the entire forward pass, needed for backpropagation
For a model like GPT-3, training requires over 1 TB of memory just for the optimizer states (if using Adam). This is why training is done with thousands of GPUs, each holding a slice of the model. Inference, on the other hand, can be done with a single GPU if the model fits in its memory, but as models grow, even inference requires multiple GPUs or specialized hardware with large HBM stacks.
Why Memory Capacity Matters for Performance
Memory capacity directly affects the batch size and context length an AI system can handle. A larger batch size increases throughput (more queries processed per second), but requires more memory to store all the intermediate data. Similarly, a longer context window (the amount of text the model can consider at once) requires a larger KV cache. If memory runs out, the system must reduce batch size or context length, which degrades performance or limits functionality.
For example, a model with a 128K token context window might need tens of GB of KV cache per conversation. To serve many users simultaneously, the data center must have enough total memory across all GPUs to hold all active conversations’ caches. This is why memory capacity is a key purchasing criterion for AI servers.
Common Misconception: More Memory = More Speed?
It’s tempting to think that adding more memory automatically makes AI faster, but that’s not true. The bottleneck is often memory bandwidth, not capacity. Even if you have 1 TB of memory, if the bandwidth is only 100 GB/s, the GPU will starve. Conversely, having too much capacity with high bandwidth is wasteful if the model doesn’t need it. The right balance depends on the workload. For inference, bandwidth is usually the limiting factor; for training, capacity is often the constraint because you need to store all the gradients and activations.
Frequently Asked Questions
Why not just use cloud storage for AI models?
Cloud storage (like object storage) is for persistent files, not for active computation. The latency to fetch data from a remote server is milliseconds, while a GPU needs data in nanoseconds. Even with high-speed networking, it’s too slow. That’s why models must be loaded into local memory before inference.
How much memory does a typical AI data center have?
It varies widely. A single server with 8 GPUs might have 8 × 80 GB = 640 GB of HBM, plus 1-2 TB of system RAM. A large cluster with thousands of GPUs could have petabytes of total memory. But the key is the per-GPU memory, which determines what models can run.
Can AI models run without HBM?
Yes, but with trade-offs. Some models can run on regular DDR4/DDR5 memory using CPU inference, but they are much slower. For example, a 70B parameter model might run on a server with 256 GB of RAM, but inference could take minutes per query instead of seconds. HBM is essential for real-time applications.
Will memory requirements keep growing?
Yes, as models get larger and context windows expand, memory needs increase. However, researchers are developing techniques like quantization (using fewer bits per weight) and model compression to reduce memory usage. Still, the trend is toward larger models, so memory demand is likely to rise.
What is the difference between VRAM and HBM?
VRAM is a general term for video memory, which includes GDDR6 and HBM. HBM is a specific type of VRAM that is stacked vertically to achieve higher bandwidth and lower power consumption. GDDR6 is more common in consumer GPUs, while HBM is used in data center accelerators like NVIDIA’s A100 and H100.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.