ThisCalculator

Everyday

Monitor NVIDIA GPUs with Prometheus and Grafana

Updated · Standard formula, shown below

This guide walks you through setting up full monitoring for NVIDIA GPUs using Prometheus for metric collection and Grafana for visualization. You’ll end up with live dashboards showing GPU temperature, utilization, memory usage, power draw, and more — all from a single host or a cluster. You’ll need a Linux server (or Windows with WSL) with an NVIDIA GPU, plus basic familiarity with the command line.

What You’ll Need

  • A machine with an NVIDIA GPU and the NVIDIA driver installed (check with nvidia-smi).
  • Docker and Docker Compose (recommended for easy setup) — or you can install Prometheus, Grafana, and the exporter directly.
  • Port 9100 (node exporter) and 9400 (GPU exporter) available if you want to monitor system metrics too.
  • Basic knowledge of editing YAML files and running shell commands.

Step-by-Step

  1. Install the NVIDIA GPU Exporter
    The official exporter from NVIDIA is nvidia_gpu_exporter (also known as DCGM exporter). The simplest way is to run it as a container. Create a directory for your monitoring stack and add this docker-compose.yml:
version: '3.8'
services:
  gpu-exporter:
    image: nvidia/dcgm-exporter:latest
    runtime: nvidia
    environment:
      - DCGM_EXPORTER_LISTEN=":9400"
    ports:
      - "9400:9400"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

If you’re not using Docker, download the binary from the NVIDIA/dcgm-exporter GitHub releases and run it with ./dcgm-exporter. Verify it’s working by visiting http://localhost:9400/metrics — you should see lines starting with DCGM_.

  1. Set Up Prometheus
    Create a prometheus.yml configuration file. Add a scrape job for the GPU exporter. If you’re using Docker Compose, add Prometheus as a service:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'

In prometheus.yml, add the scrape config:

scrape_configs:
  - job_name: 'gpu'
    static_configs:
      - targets: ['gpu-exporter:9400']

If you’re running Prometheus directly on the host, use localhost:9400 instead. Start the stack with docker-compose up -d and check http://localhost:9090/targets — the gpu target should be UP.

  1. Install Grafana
    Add Grafana to your docker-compose.yml:
  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - grafana-data:/var/lib/grafana
volumes:
  grafana-data:

Restart the stack. Open http://localhost:3000, log in with admin/admin (you’ll be prompted to change the password).

  1. Add Prometheus as a Data Source
    In Grafana, go to Configuration → Data Sources → Add data source. Choose Prometheus. Set the URL to http://prometheus:9090 (or http://localhost:9090 if Grafana is on the host). Click Save & Test — you should see a green success message.
  1. Import a GPU Dashboard
    Go to Dashboards → Import. Enter dashboard ID 12239 (a popular NVIDIA DCGM dashboard) and click Load. Select your Prometheus data source and click Import. You’ll immediately see panels for GPU utilization, temperature, memory, power, and more. If you prefer to build your own, create a new dashboard and add panels with queries like DCGM_FI_DEV_GPU_UTIL, DCGM_FI_DEV_MEM_TEMP, or DCGM_FI_DEV_POWER_USAGE.
  1. Optional: Add Node Exporter for System Metrics
    To see CPU, RAM, and disk alongside GPU stats, add the node exporter:
  node-exporter:
    image: prom/node-exporter:latest
    ports:
      - "9100:9100"

Add a scrape job in prometheus.yml for node-exporter:9100, then re-import or extend your dashboard with system panels.

  1. Set Up Alerting (Optional)
    In Grafana, go to Alerting → Contact points to add email, Slack, or webhook notifications. Create alert rules for high GPU temperature or low memory. Prometheus can also send alerts via Alertmanager, but Grafana’s built-in alerting is easier for beginners.

Common Mistakes

  • Forgetting the NVIDIA runtime — If the DCGM exporter container fails to start, it’s almost always because the Docker runtime isn’t set to nvidia or the NVIDIA Container Toolkit isn’t installed.
  • Wrong target address — Using localhost inside a Docker container refers to the container itself, not the host. Use service names (like gpu-exporter) in Docker networks.
  • Scraping the wrong port — The default DCGM exporter port is 9400, not 9100. Double-check your port mapping.
  • Not restarting after config changes — Prometheus and Grafana don’t auto-reload config files. Run docker-compose restart prometheus after editing prometheus.yml.
  • Using outdated dashboard IDs — Some dashboard IDs may be deleted or changed. If import fails, search for “NVIDIA DCGM” in the Grafana dashboard library.

Tips

  • Use labels — If you have multiple GPUs, the exporter adds a gpu label (e.g., gpu="0"). Use this in Grafana variables to filter by GPU.
  • Persist Grafana data — Always map a volume for Grafana’s data directory so dashboards survive container restarts.
  • Monitor multiple hosts — Run the DCGM exporter on each GPU server and add all their IPs to the targets list in Prometheus.
  • Set retention — Prometheus stores metrics for 15 days by default. If you need longer, add --storage.tsdb.retention.time=30d to the Prometheus command.
  • Use Grafana variables — Create a variable for instance or gpu to make your dashboard dynamic.

Troubleshooting

  • Exporter container won’t start — Check the logs with docker logs gpu-exporter. If it says “NVIDIA driver not detected”, install the NVIDIA Container Toolkit and restart Docker. Ensure the driver is installed on the host.
  • No metrics at /metrics — Try curl localhost:9400/metrics on the host. If empty, the exporter may be listening on a different interface. Use --listen-address=:9400 explicitly.
  • Prometheus target shows DOWN — Verify the target address is reachable from the Prometheus container: docker exec prometheus wget -qO- http://gpu-exporter:9400/metrics (if wget is available). Check for typos in the YAML.
  • Grafana shows “No data” — Ensure the data source URL is correct and that Prometheus has scraped at least one sample. In Prometheus, go to Status → Targets to confirm the endpoint is UP. Then in Grafana, check the query editor for the metric name — it must match exactly (e.g., DCGM_FI_DEV_GPU_UTIL).
  • Dashboard panels are empty — Some dashboards expect a specific job name like dcgm. Edit the panel query and change the job label to match your scrape job name (e.g., job="gpu").
  • High cardinality warnings — If you have many GPUs, you may see performance issues. Reduce scrape frequency or filter metrics with metric_relabel_configs in Prometheus.

Frequently Asked Questions

Can I monitor NVIDIA GPUs on Windows?

Yes, but the DCGM exporter is Linux-only. You can run it inside a Linux VM or use WSL2 with Docker. Alternatively, use the Windows exporter like nvidia_gpu_exporter from the Prometheus community, but it’s less maintained.

What’s the difference between DCGM and nvidia-smi?

nvidia-smi is a command-line tool for quick checks. DCGM (Data Center GPU Manager) provides a richer set of metrics (like memory bandwidth, PCIe throughput) and is designed for continuous monitoring. The exporter exposes DCGM metrics in Prometheus format.

How do I monitor GPU memory usage?

Use the metric DCGM_FI_DEV_FB_USED (framebuffer memory used) or DCGM_FI_DEV_MEM_COPY_UTIL for memory copy utilization. The dashboard 12239 includes these.

Can I monitor multiple GPU servers with one Prometheus?

Yes. Add each server’s IP and port to the targets list in the scrape config. Use labels like instance to differentiate them in Grafana.

How do I alert when a GPU overheats?

In Grafana, create an alert rule on the DCGM_FI_DEV_GPU_TEMP metric with a threshold (e.g., >85°C). Set a contact point for email or Slack. Prometheus Alertmanager is another option but requires extra setup.

Is there a way to monitor GPU power consumption?

Yes, the metric DCGM_FI_DEV_POWER_USAGE gives instantaneous power in watts. The dashboard includes a power panel. Note that power readings may be unavailable on some GPUs or virtual machines.

Related

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.