Estimate AI Serving Capacity with Limited Accelerators

Updated 5 November 2026 · Standard formula, shown below

This guide shows you how to estimate how many AI inference requests your serving infrastructure can handle when you have a fixed, limited number of accelerators (GPUs, TPUs, or custom ASICs). You’ll learn to calculate throughput and latency based on your model’s characteristics, hardware specs, and traffic patterns, so you can plan capacity, set expectations, and decide whether you need to optimize the model, the hardware, or the deployment.

What You’ll Need

Model details: number of parameters, input/output token sizes (for LLMs) or input dimensions (for vision/other models), and the framework you use (PyTorch, TensorFlow, ONNX Runtime, etc.).
Hardware specs: total memory (VRAM), memory bandwidth, compute throughput (FLOPS), and the number of accelerators you have.
Benchmark data from your own environment: latency per request (or per token) and throughput at a given batch size. If you don’t have this, you’ll need to run a quick benchmark or use published numbers for a similar model/hardware.
Traffic profile: expected requests per second (RPS), average request size (tokens or pixels), and peak vs. average load.
A spreadsheet or a simple script to do the arithmetic.

Step-by-Step

Define your serving objective. Decide whether you need to meet a target latency (e.g., p95 < 500 ms) or a target throughput (e.g., 100 requests per second). This determines which metric you'll optimize for.
Measure or estimate per-request latency. Run a benchmark with a representative workload on a single accelerator. For LLMs, measure time to first token (TTFT) and time per output token (TPOT). For other models, measure end-to-end latency for a single request at batch size 1.
Determine the maximum batch size that fits in memory. Use the model size and activation memory to estimate. For LLMs, a rough formula is: memory â‰ˆ (model parameters Ã— bytes per param) + (batch size Ã— sequence length Ã— layers Ã— hidden size Ã— bytes per activation). Use your framework’s memory profiler if available.
Benchmark throughput at different batch sizes. Run inference with batch sizes 1, 2, 4, 8, 16, etc. Measure the throughput (requests per second) and latency per request. You’ll see that throughput increases with batch size, but latency per request also increases due to queueing.
Find the sweet spot. Plot throughput vs. latency. Choose the batch size that gives the highest throughput while still meeting your latency target. This is your per-accelerator capacity.
Multiply by the number of accelerators. If you have N identical accelerators and they are independent (no shared bottleneck), your total capacity is roughly N Ã— per-accelerator throughput. However, account for overhead: if you use a distributed serving system, there’s a small loss (typically 5-10%) due to communication and scheduling.
Apply a utilization factor. Real-world traffic is bursty. To avoid queueing delays, plan for a peak utilization of 60-80% of theoretical capacity. For example, if your theoretical capacity is 200 RPS, plan for 120-160 RPS at peak.
Validate with a load test. Use a load testing tool (e.g., k6, Locust, or a simple script) to send synthetic traffic to your serving endpoint. Measure actual latency and throughput under load. Adjust your batch size and concurrency settings based on results.
Plan for headroom. Leave some accelerator capacity for retries, spikes, and model updates. A common practice is to keep at least 20% headroom above your expected peak.

Common Mistakes

Using theoretical peak FLOPS. Real-world inference rarely achieves peak compute utilization. Always use measured numbers from your actual workload.
Ignoring memory bandwidth. For autoregressive models (like LLMs), memory bandwidth is often the bottleneck, not compute. Check if your model is memory-bound or compute-bound.
Assuming linear scaling. Adding more accelerators doesn’t always double capacity due to communication overhead, especially for small batch sizes.
Forgetting about the serving framework overhead. The framework (e.g., TensorFlow Serving, Triton, vLLM) adds latency and reduces throughput. Include that in your benchmarks.
Using average latency instead of percentile. Users care about p95 or p99 latency. Always measure the tail, not just the mean.

Tips

Use continuous batching. For LLMs, frameworks like vLLM or TensorRT-LLM can dynamically batch requests, increasing throughput significantly. Benchmark with the same framework you plan to use.
Quantize the model. Reducing precision (e.g., from FP16 to INT8) can reduce memory and increase throughput, but may affect accuracy. Test on your data.
Cache results. If you have repeated requests (e.g., same prompt), caching can drastically reduce load. Implement a simple cache and measure the hit rate.
Monitor in production. Use metrics like GPU utilization, memory usage, and request queue length to detect when you’re approaching capacity.
Consider autoscaling. If your infrastructure allows, add accelerators dynamically during peak times. This is easier with cloud-based accelerators.

Troubleshooting

Your benchmark shows lower throughput than expected. Check if the model is using all available memory bandwidth. Try increasing batch size or using a framework with better batching. Also, check if the CPU is a bottleneck (e.g., data preprocessing or tokenization).
Latency is too high at the target batch size. Reduce the batch size, even if it lowers throughput. Alternatively, optimize the model (e.g., use a smaller variant, prune, or distill).
Out-of-memory errors during benchmarking. Reduce batch size, sequence length, or use gradient checkpointing if training. For inference, try memory-efficient attention (e.g., FlashAttention) or offload some layers to CPU.
Scaling across multiple accelerators doesn’t work as expected. Ensure your serving framework is configured for multi-GPU. Check for network bottlenecks (e.g., PCIe vs. NVLink) and consider using tensor parallelism for large models.
Your capacity estimate doesn’t match production. Production traffic is more varied. Re-run your load test with a more realistic mix of request sizes and arrival patterns. Also, consider that some requests may be retried, adding extra load.

Frequently Asked Questions

How do I estimate capacity without any existing benchmarks?

Use published benchmarks for a similar model on the same hardware (e.g., from MLPerf or vendor blogs). Adjust for your model size and framework. Then run a small-scale test to validate.

What is the difference between throughput and latency in this context?

Throughput is the number of requests processed per second. Latency is the time it takes to process a single request. They are often traded off: increasing batch size boosts throughput but increases latency per request.

How do I account for different request sizes?

Measure the average request size (e.g., average number of tokens) and use that in your benchmarks. For LLMs, longer outputs increase latency and memory. If your traffic is highly variable, test with a mix of sizes.

Should I use a GPU or a CPU for serving?

For most modern AI models, GPUs or specialized accelerators are necessary to meet latency requirements. CPUs can work for very small models or low traffic, but you’ll get much lower throughput per dollar.

Can I use the same estimate for training capacity?

No. Training is compute-intensive and uses different patterns (e.g., backpropagation). This guide is for inference only. Training capacity depends on batch size, sequence length, and number of steps, and is usually measured in samples per second.

What if I have a mix of different accelerator types?

Estimate capacity per type separately, then sum. Be aware that the slowest type may become a bottleneck if you route traffic unevenly. Use a load balancer that considers each accelerator’s capacity.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.