How to Benchmark Ai Inference Hardware for Latency Throughput and Cost
This guide walks you through benchmarking AI inference hardware to measure latency, throughput, and cost per inference. You’ll learn how to design a fair test, collect reliable numbers, and compare options for your workload. You’ll need access to the hardware, a representative model, and a basic understanding of the command line.
What You’ll Need
- A machine with the target hardware (GPU, CPU, or accelerator) and drivers installed.
- A deep learning framework (e.g., PyTorch, TensorFlow) and the model you plan to deploy, or a representative model from a model zoo.
- A benchmarking tool or script. Options include
triton(for NVIDIA GPUs),onnxruntimebenchmarks, or a simple custom Python script. - A way to measure power consumption (e.g.,
nvidia-smifor GPUs,perffor CPUs) if you want cost per inference. - Cloud provider pricing sheets if you’re comparing cloud instances.
Step-by-Step
- Define your workload. Decide on the model, input size, batch size, and precision (FP32, FP16, INT8). These must be identical across all hardware you compare.
- Set up the environment. Install the same framework version and dependencies on each machine. Use a virtual environment or container to keep things consistent.
- Warm up the hardware. Run a few inference passes before measuring to allow the GPU/CPU to reach steady-state temperature and trigger any caching.
- Measure latency. Run a single inference (batch size 1) multiple times (e.g., 1000 runs) and record the time per inference. Calculate the average, median, and 95th percentile latency. Use a high-resolution timer.
- Measure throughput. Send a continuous stream of requests (or use a large batch) and measure how many inferences are completed per second. Tools like
perf_analyzer(Triton) orloadgen(MLPerf) can automate this. - Record power consumption. While running the throughput test, sample power draw (e.g.,
nvidia-smi --query-gpu=power.draw --format=csvevery 100ms). Average the readings. - Calculate cost per inference. For cloud hardware: divide the hourly instance price by the throughput (inferences per hour) to get cost per inference. For on-prem: estimate hardware cost divided by its lifespan, plus electricity cost (power draw in kW * hours * electricity rate).
- Repeat across batch sizes. Test batch sizes of 1, 8, 32, and 64 to understand how latency and throughput scale. Larger batches usually improve throughput but increase latency.
- Test with realistic input sizes. If your production data has variable input sizes, include those in the benchmark. Use a representative distribution.
- Document everything. Record the exact software versions, model weights, input data, and hardware specs. This ensures reproducibility.
Common Mistakes
- Not warming up. Cold-start inferences are slower and will skew latency results.
- Using different input sizes. Comparing a 224×224 image on one GPU and a 512×512 on another invalidates the comparison.
- Ignoring power throttling. Some GPUs throttle after sustained load, reducing performance. Run long enough to observe this.
- Measuring latency with a batch size >1. Latency per inference is not the same as total time divided by batch size; it’s the time for the whole batch to complete.
- Forgetting to pin CPU threads. For CPU benchmarking, thread pinning can significantly affect results.
Tips
- Use a benchmark tool that supports both latency and throughput modes, like
perf_analyzerorghbench. - For cloud instances, choose the same instance type family (e.g., A10 vs. A100) to isolate the GPU’s impact.
- Check if the hardware supports TensorRT or OpenVINO optimizations; these can change results dramatically.
- Run the benchmark at least three times and take the median to reduce variance.
- If you’re benchmarking for a real-time application, focus on the 95th or 99th percentile latency, not just the average.
Troubleshooting
- Results vary wildly between runs. Ensure no other processes are using the GPU/CPU. Use
nvidia-smito check GPU utilization. Consider using a dedicated machine. - Throughput drops after a few minutes. This is likely thermal throttling. Check temperatures and improve cooling, or note the steady-state throughput.
- Latency is higher than expected. Check if the model is using dynamic shapes or if the framework is doing graph optimization on the first call. Warm up longer.
- Cost per inference seems too high. Double-check your throughput units (e.g., inferences per second vs. per hour). Also, ensure you’re using the correct instance price (on-demand vs. spot).
- Benchmark tool crashes. Reduce the number of concurrent requests or use a smaller batch size. Check memory usage.
Frequently Asked Questions
What is the difference between latency and throughput?
Latency is the time to complete one inference (usually measured in milliseconds). Throughput is the number of inferences completed per second. They are related but not directly proportional—increasing batch size often increases throughput but also increases latency.
What is a good batch size for benchmarking?
It depends on your use case. For real-time applications (e.g., a chatbot), batch size 1 is common. For offline processing (e.g., image recognition on a dataset), batch sizes of 32-64 are typical. Benchmark multiple sizes to see the trade-off.
Should I use FP16 or INT8?
Lower precision (FP16, INT8) generally runs faster and uses less memory, but may reduce accuracy. If your model can tolerate the accuracy loss, benchmark these to see the performance gain. Always validate accuracy after quantization.
How do I measure power consumption on a CPU?
Use tools like perf with the power events, or turbostat on Linux. Alternatively, use a hardware power meter at the wall outlet for the whole system.
Can I benchmark without a dedicated tool?
Yes, you can write a simple Python script that times inference calls using time.perf_counter(). For throughput, you can run multiple threads or processes to send requests concurrently. However, dedicated tools often provide more accurate and reproducible results.
How often should I re-benchmark?
Re-benchmark whenever you change the model, framework version, driver, or hardware. Even minor software updates can affect performance. For cloud instances, it’s good practice to re-run benchmarks periodically as the underlying hardware may be updated.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.