Benchmark AI Inference Hardware Beyond Peak TOPS

Updated 14 November 2026 · Standard formula, shown below

Peak TOPS (trillions of operations per second) is a marketing number that rarely reflects real-world AI inference speed. This guide shows you how to measure actual performance using open-source tools and realistic workloads, so you can compare hardware for your specific models, batch sizes, and latency requirements.

What You’ll Need

A machine with the AI accelerator you want to test (GPU, NPU, or CPU with AVX-512/VNNI).
Python 3.8+ installed, with pip.
A model to test — pick one that matches your use case (e.g., ResNet-50 for image classification, BERT-base for NLP).
Benchmarking library: onnxruntime (cross-platform), tensorflow or pytorch with their built-in profilers, or mlperf-inference for standardized suites.
A way to measure power draw (optional but recommended): a watt meter at the wall or nvidia-smi / rocm-smi for on-chip power.

Step-by-Step

Define your metric. Decide whether you care about throughput (inferences per second) or latency (time per inference). For batch processing, throughput matters; for real-time applications, latency at a given batch size (usually 1) is critical.
Set up the environment. Create a fresh Python virtual environment and install the inference runtime for your hardware. For example: pip install onnxruntime-gpu for NVIDIA GPUs, onnxruntime for CPU, or tensorflow-cpu for CPU-only. Ensure CUDA/cuDNN or ROCm are correctly installed if using GPU.
Export your model to ONNX. If you have a PyTorch or TensorFlow model, convert it to ONNX using torch.onnx.export or the TF-ONNX converter. This gives you a hardware-agnostic format that most runtimes can load. Use a fixed input shape (e.g., 1x3x224x224 for ResNet) to avoid dynamic-shape overhead.
Write a benchmark script. The core loop should: load the model, warm up with a few dummy inputs (e.g., 10 iterations), then time N iterations (e.g., 100) using time.perf_counter(). Record both total time and per-iteration latency. For throughput, run several batches and divide by total time.
Measure latency correctly. For latency, use a batch size of 1 and measure the time for each inference. Take the median (not mean) to avoid outlier spikes. Report p50, p90, and p99 percentiles for a realistic picture.
Measure throughput. Increase batch size (e.g., 1, 8, 32, 64) and measure inferences per second (batch size / time per batch). Plot throughput vs. batch size to find the saturation point.
Run multiple trials. Run each configuration at least 5 times, and take the best or median result. Hardware throttling and background processes can skew single runs.
Monitor power and temperature. Use nvidia-smi dmon or a wall meter to record average power during the benchmark. Compute performance per watt (inferences per second per watt) — this is often more important than raw speed, especially for edge devices.
Compare with peak TOPS. Calculate the achieved TOPS by multiplying inferences per second by the number of operations per inference (e.g., ResNet-50 ~ 7.7 GFLOPs, so 1000 inferences/s = 7.7 TFLOPs). Divide by the peak TOPS to get utilization percentage — usually 10-30% for real workloads.
Test with realistic input sizes. If your application uses 1080p images, don’t benchmark with 224×224. Use your actual input dimensions, because memory bandwidth and compute patterns change significantly.

Common Mistakes

Using only batch size 1. This underutilizes the hardware and gives a misleadingly low throughput number. Always test a range of batch sizes.
Ignoring warm-up. The first few inferences include model loading and kernel compilation. Skipping warm-up inflates latency numbers.
Measuring in a noisy environment. Other processes, thermal throttling, or power management can skew results. Close background apps and disable CPU frequency scaling if possible.
Comparing different frameworks. PyTorch vs. ONNX Runtime vs. TensorRT can vary by 2-3x. Always use the same runtime and optimization level when comparing hardware.
Not checking precision. FP16 vs. FP32 vs. INT8 can change speed dramatically. Decide which precision your production model uses and benchmark that.
Assuming TOPS is achievable. Peak TOPS assumes perfect compute utilization with no memory stalls — real models rarely exceed 30% of that.

Tips

Use a standardized model like ResNet-50 or BERT-base so you can compare your results with published benchmarks.
Enable graph optimizations in ONNX Runtime: sess_options.graph_optimization_level = GraphOptimizationLevel.ORT_ENABLE_ALL.
For GPUs, use TensorRT if available — it can double throughput compared to stock ONNX Runtime.
Record the exact software versions (CUDA, cuDNN, runtime) — they dramatically affect results.
Test with your actual data distribution. If your images are 4K, don’t use ImageNet’s 224×224.
Use a script that outputs a JSON report for easy comparison across runs.

Troubleshooting

Results are wildly inconsistent. Check for thermal throttling — monitor temperature during the run. Also ensure no other heavy processes are running.
ONNX Runtime fails to load the model. Verify the model is exported correctly with onnx.checker.check_model. Also ensure the opset version is compatible with your runtime.
GPU not being used. Run nvidia-smi during the benchmark. If GPU utilization is 0%, your runtime might be falling back to CPU. Check that you installed the GPU version of onnxruntime and that CUDA is visible to Python.
Out of memory. Reduce batch size or use a smaller input resolution. For large models, enable memory optimization in the runtime.
Latency spikes every few iterations. This could be garbage collection in Python. Use gc.disable() during timing, and allocate input buffers once outside the loop.
Power readings seem off. Wall meters measure the whole system, not just the accelerator. Use on-chip sensors (nvidia-smi) for the GPU alone.

Frequently Asked Questions

Why is my real-world performance so much lower than peak TOPS?

Peak TOPS assumes perfectly parallel compute with no memory bottlenecks. Real models have dependencies, memory access patterns, and overhead that limit utilization to typically 10-30% of peak.

Should I use batch size 1 or larger for benchmarking?

It depends on your use case. For real-time applications (e.g., object detection on a video stream), batch size 1 latency is what matters. For offline processing (e.g., batch image classification), larger batches improve throughput.

What is the difference between latency and throughput?

Latency is the time for one inference (usually milliseconds). Throughput is the number of inferences per second (IPS). They are related but not inversely proportional due to batching overhead.

Can I trust published TOPS numbers for comparing hardware?

Only as a rough upper bound. Different vendors measure TOPS at different precisions (INT8 vs FP16) and with different sparsity assumptions. Always benchmark with your own model and runtime.

How do I measure performance per watt?

Measure the average power draw during a steady-state benchmark (after warm-up) and divide the throughput by that power. For GPUs, use nvidia-smi --query-gpu=power.draw.

Should I use a benchmarking suite like MLPerf?

MLPerf is excellent for standardized comparisons, but it’s heavy to set up and may not reflect your specific workload. For quick internal comparisons, a custom script is often sufficient.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.