Monitor NVIDIA GPUs with Prometheus and Grafana
This guide walks you through setting up full monitoring for NVIDIA GPUs using Prometheus for metric collection and Grafana for visualization. You’ll end up with live dashboards showing GPU temperature, utilization, memory usage, power draw, and more — all from a single host or a cluster. You’ll need a Linux server (or Windows with WSL) with an NVIDIA GPU, plus basic familiarity with the command line.
What You’ll Need
- A machine with an NVIDIA GPU and the NVIDIA driver installed (check with
nvidia-smi). - Docker and Docker Compose (recommended for easy setup) — or you can install Prometheus, Grafana, and the exporter directly.
- Port 9100 (node exporter) and 9400 (GPU exporter) available if you want to monitor system metrics too.
- Basic knowledge of editing YAML files and running shell commands.
Step-by-Step
- Install the NVIDIA GPU Exporter
The official exporter from NVIDIA isnvidia_gpu_exporter(also known as DCGM exporter). The simplest way is to run it as a container. Create a directory for your monitoring stack and add thisdocker-compose.yml:
version: '3.8'
services:
gpu-exporter:
image: nvidia/dcgm-exporter:latest
runtime: nvidia
environment:
- DCGM_EXPORTER_LISTEN=":9400"
ports:
- "9400:9400"
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
If you’re not using Docker, download the binary from the NVIDIA/dcgm-exporter GitHub releases and run it with ./dcgm-exporter. Verify it’s working by visiting http://localhost:9400/metrics — you should see lines starting with DCGM_.
- Set Up Prometheus
Create aprometheus.ymlconfiguration file. Add a scrape job for the GPU exporter. If you’re using Docker Compose, add Prometheus as a service:
prometheus:
image: prom/prometheus:latest
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
ports:
- "9090:9090"
command:
- '--config.file=/etc/prometheus/prometheus.yml'
In prometheus.yml, add the scrape config:
scrape_configs:
- job_name: 'gpu'
static_configs:
- targets: ['gpu-exporter:9400']
If you’re running Prometheus directly on the host, use localhost:9400 instead. Start the stack with docker-compose up -d and check http://localhost:9090/targets — the gpu target should be UP.
- Install Grafana
Add Grafana to yourdocker-compose.yml:
grafana:
image: grafana/grafana:latest
ports:
- "3000:3000"
environment:
- GF_SECURITY_ADMIN_PASSWORD=admin
volumes:
- grafana-data:/var/lib/grafana
volumes:
grafana-data:
Restart the stack. Open http://localhost:3000, log in with admin/admin (you’ll be prompted to change the password).
- Add Prometheus as a Data Source
In Grafana, go to Configuration → Data Sources → Add data source. Choose Prometheus. Set the URL tohttp://prometheus:9090(orhttp://localhost:9090if Grafana is on the host). Click Save & Test — you should see a green success message.
- Import a GPU Dashboard
Go to Dashboards → Import. Enter dashboard ID12239(a popular NVIDIA DCGM dashboard) and click Load. Select your Prometheus data source and click Import. You’ll immediately see panels for GPU utilization, temperature, memory, power, and more. If you prefer to build your own, create a new dashboard and add panels with queries likeDCGM_FI_DEV_GPU_UTIL,DCGM_FI_DEV_MEM_TEMP, orDCGM_FI_DEV_POWER_USAGE.
- Optional: Add Node Exporter for System Metrics
To see CPU, RAM, and disk alongside GPU stats, add the node exporter:
node-exporter:
image: prom/node-exporter:latest
ports:
- "9100:9100"
Add a scrape job in prometheus.yml for node-exporter:9100, then re-import or extend your dashboard with system panels.
- Set Up Alerting (Optional)
In Grafana, go to Alerting → Contact points to add email, Slack, or webhook notifications. Create alert rules for high GPU temperature or low memory. Prometheus can also send alerts via Alertmanager, but Grafana’s built-in alerting is easier for beginners.
Common Mistakes
- Forgetting the NVIDIA runtime — If the DCGM exporter container fails to start, it’s almost always because the Docker runtime isn’t set to
nvidiaor the NVIDIA Container Toolkit isn’t installed. - Wrong target address — Using
localhostinside a Docker container refers to the container itself, not the host. Use service names (likegpu-exporter) in Docker networks. - Scraping the wrong port — The default DCGM exporter port is 9400, not 9100. Double-check your port mapping.
- Not restarting after config changes — Prometheus and Grafana don’t auto-reload config files. Run
docker-compose restart prometheusafter editingprometheus.yml. - Using outdated dashboard IDs — Some dashboard IDs may be deleted or changed. If import fails, search for “NVIDIA DCGM” in the Grafana dashboard library.
Tips
- Use labels — If you have multiple GPUs, the exporter adds a
gpulabel (e.g.,gpu="0"). Use this in Grafana variables to filter by GPU. - Persist Grafana data — Always map a volume for Grafana’s data directory so dashboards survive container restarts.
- Monitor multiple hosts — Run the DCGM exporter on each GPU server and add all their IPs to the
targetslist in Prometheus. - Set retention — Prometheus stores metrics for 15 days by default. If you need longer, add
--storage.tsdb.retention.time=30dto the Prometheus command. - Use Grafana variables — Create a variable for
instanceorgputo make your dashboard dynamic.
Troubleshooting
- Exporter container won’t start — Check the logs with
docker logs gpu-exporter. If it says “NVIDIA driver not detected”, install the NVIDIA Container Toolkit and restart Docker. Ensure the driver is installed on the host. - No metrics at
/metrics— Trycurl localhost:9400/metricson the host. If empty, the exporter may be listening on a different interface. Use--listen-address=:9400explicitly. - Prometheus target shows DOWN — Verify the target address is reachable from the Prometheus container:
docker exec prometheus wget -qO- http://gpu-exporter:9400/metrics(if wget is available). Check for typos in the YAML. - Grafana shows “No data” — Ensure the data source URL is correct and that Prometheus has scraped at least one sample. In Prometheus, go to Status → Targets to confirm the endpoint is UP. Then in Grafana, check the query editor for the metric name — it must match exactly (e.g.,
DCGM_FI_DEV_GPU_UTIL). - Dashboard panels are empty — Some dashboards expect a specific job name like
dcgm. Edit the panel query and change the job label to match your scrape job name (e.g.,job="gpu"). - High cardinality warnings — If you have many GPUs, you may see performance issues. Reduce scrape frequency or filter metrics with
metric_relabel_configsin Prometheus.
Frequently Asked Questions
Can I monitor NVIDIA GPUs on Windows?
Yes, but the DCGM exporter is Linux-only. You can run it inside a Linux VM or use WSL2 with Docker. Alternatively, use the Windows exporter like nvidia_gpu_exporter from the Prometheus community, but it’s less maintained.
What’s the difference between DCGM and nvidia-smi?
nvidia-smi is a command-line tool for quick checks. DCGM (Data Center GPU Manager) provides a richer set of metrics (like memory bandwidth, PCIe throughput) and is designed for continuous monitoring. The exporter exposes DCGM metrics in Prometheus format.
How do I monitor GPU memory usage?
Use the metric DCGM_FI_DEV_FB_USED (framebuffer memory used) or DCGM_FI_DEV_MEM_COPY_UTIL for memory copy utilization. The dashboard 12239 includes these.
Can I monitor multiple GPU servers with one Prometheus?
Yes. Add each server’s IP and port to the targets list in the scrape config. Use labels like instance to differentiate them in Grafana.
How do I alert when a GPU overheats?
In Grafana, create an alert rule on the DCGM_FI_DEV_GPU_TEMP metric with a threshold (e.g., >85°C). Set a contact point for email or Slack. Prometheus Alertmanager is another option but requires extra setup.
Is there a way to monitor GPU power consumption?
Yes, the metric DCGM_FI_DEV_POWER_USAGE gives instantaneous power in watts. The dashboard includes a power panel. Note that power readings may be unavailable on some GPUs or virtual machines.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.