GPU Memory & Interconnects for AI Inference

Updated 6 November 2026 · Standard formula, shown below

This guide explains how to size GPU memory (VRAM) and server interconnects (PCIe, NVLink, InfiniBand) for private AI inference workloads. You’ll learn the key formulas, practical rules of thumb, and common pitfalls, so you can pick hardware that fits your model, batch size, and latency targets without overspending.

What You’ll Need

Model details: parameter count (e.g., 7B, 70B), precision (FP16, FP8, INT8), and context length (for LLMs).
Framework knowledge: PyTorch, TensorRT, vLLM, or similar—each has slightly different memory overhead.
Workload profile: expected concurrent requests, batch size, and acceptable latency.
Hardware specs: GPU memory bandwidth, interconnect topology (PCIe lanes, NVLink bridges, InfiniBand ports).

Step-by-Step

Estimate model weights memory. Multiply the number of parameters by the bytes per parameter. For FP16 (2 bytes), a 7B model needs ~14 GB; for INT8 (1 byte), ~7 GB. Use the formula: Weights (GB) = Params (billions) × Bytes per param.
Add activation memory. During inference, activations (intermediate tensors) consume memory proportional to batch size, sequence length, and hidden dimensions. A rough estimate: for transformers, activations ≈ 2 × (batch size × sequence length × hidden size) × bytes per element. For most LLM inference, this is 20–40% of weight memory for moderate batch sizes.
Include framework overhead. CUDA contexts, kernel workspaces, and framework caches add 1–2 GB per GPU. Also reserve ~10% headroom for fragmentation and temporary buffers.
Calculate total VRAM per GPU. Total = Weights + Activations + Overhead. For a 7B FP16 model with batch 32 and sequence 2048, activations might be ~4 GB, so total ≈ 14 + 4 + 2 = 20 GB. Choose a GPU with at least that much VRAM (e.g., 24 GB).
Decide if you need multiple GPUs. If the model fits on one GPU, you generally don’t need inter-GPU interconnects beyond standard PCIe. Only when the model exceeds a single GPU’s VRAM do you need tensor parallelism (splitting layers) or pipeline parallelism.
Choose interconnect for multi-GPU. For tensor parallelism, every forward pass requires all-to-all communication between GPUs. NVLink (or NVSwitch) provides ~600 GB/s per GPU, while PCIe 4.0 x16 offers ~32 GB/s. If your workload is latency-sensitive, use NVLink; if throughput is less critical, PCIe may suffice.
Size server interconnects (InfiniBand). For multi-node inference, use InfiniBand (e.g., HDR 200 Gb/s) or high-speed Ethernet with RDMA. Rule of thumb: each GPU should have at least 25 Gb/s of network bandwidth for model-parallel inference; 100 Gb/s per GPU is safer for large models.
Validate with a benchmark. Run your actual model with representative input sizes and measure memory usage (nvidia-smi) and latency. Adjust batch size or precision if you hit memory limits.

Common Mistakes

Ignoring activation memory—especially for long sequences or large batches, activations can dominate VRAM.
Assuming FP16 uses exactly 2 bytes per parameter—some frameworks use mixed precision with additional overhead.
Overlooking CUDA context memory—each GPU process reserves ~500 MB–1 GB before loading any model.
Choosing PCIe for tensor parallelism—the communication bottleneck severely reduces speedup.
Forgetting that interconnects also carry data for data-parallel inference (e.g., gradient sync during training, but for pure inference it’s less critical).

Tips

Use quantization (INT8/FP8) to cut memory nearly in half, but verify accuracy loss is acceptable.
For LLMs, use continuous batching (e.g., vLLM) to improve GPU utilization without increasing memory proportionally.
Prefer GPUs with higher memory bandwidth (HBM2e/HBM3) for faster inference, even if VRAM is similar.
If using multiple GPUs, consider NVLink for up to 8 GPUs; beyond that, NVSwitch or InfiniBand for multi-node.
Always leave 10–20% VRAM headroom for spikes in memory usage.

Troubleshooting

Out of memory (OOM) errors: Reduce batch size, shorten sequence length, or switch to a lower precision. Check if other processes are using GPU memory.
Slow inference with multiple GPUs: Likely interconnect bottleneck. Profile communication time; if high, switch to NVLink or reduce parallelism degree.
GPU utilization low: Could be CPU bottleneck (data loading, tokenization) or small batch size. Increase batch size or use async data loading.
Incorrect memory estimate: Use nvidia-smi to see real usage; adjust your formula with framework-specific overhead.
Interconnect not detected: Check BIOS settings, driver installation, and cable seating. For NVLink, ensure GPUs are physically bridged.

Frequently Asked Questions

How much VRAM do I need for a 70B model in FP16?

Weights alone take ~140 GB. With activations and overhead, you’ll need at least 160 GB, so typically 4× 40 GB GPUs (e.g., A100) with NVLink.

Can I run a 7B model on a 16 GB GPU?

Yes, if you use INT8 quantization (weights ~7 GB) and keep batch size small. FP16 would exceed 16 GB with activations.

Is NVLink necessary for inference?

Only if you split the model across GPUs (tensor parallelism). For data-parallel inference (each GPU handles a separate request), PCIe is fine.

What’s the difference between PCIe and NVLink?

PCIe is a general-purpose bus with ~32 GB/s per x16 slot; NVLink is a GPU-to-GPU interconnect with ~300–600 GB/s, designed for low-latency communication.

How do I size InfiniBand for multi-node inference?

For each GPU, allocate 25–100 Gb/s depending on model size and parallelism. Use HDR (200 Gb/s) for large models.

Does batch size affect memory linearly?

Activations scale roughly linearly with batch size, but weights stay constant. So doubling batch size increases memory by the activation portion, not the whole model.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.