Container GPU Observability with NVIDIA DCGM Exporter & Prometheus

In production AI clusters, monitoring host-level CPU and RAM metrics is insufficient. Site Reliability Engineers (SREs) and MLOps teams must track per-container GPU VRAM allocation, GPU compute utilization percentage, power draw (Watts), thermal throttling, and hardware ECC memory errors.

Standard node_exporter does not inspect NVIDIA GPU driver telemetry. The official solution is NVIDIA DCGM (Data Center GPU Manager) Exporter, which exports low-overhead GPU metrics formatted for Prometheus scrapers.

This technical guide covers deploying DCGM Exporter in Docker, configuring Prometheus scraping, mapping metrics to Docker container names, and setting up Grafana dashboards.

---

1. What is NVIDIA DCGM Exporter?

NVIDIA DCGM (Data Center GPU Manager) is a suite of tools designed to monitor and manage NVIDIA Data Center GPUs (Tesla, A100, H100, L40S, RTX Enterprise).

[ NVIDIA Host Driver / GPU Hardware ]
         │ (NVML / DCGM Engine)
         ▼
[ dcgm-exporter Container ] (Exposes HTTP :9400 /metrics)
         │
         ▼
[ Prometheus Scraper ] ──► [ Grafana Dashboard ]

Key Metrics Tracked:

• DCGM_FI_DEV_GPU_UTIL: Overall GPU compute engine utilization (%).
• DCGM_FI_DEV_FB_USED: Framebuffer (VRAM) memory used in MB.
• DCGM_FI_DEV_POWER_USAGE: Power draw in Watts.
• DCGM_FI_DEV_GPU_TEMP: Core GPU temperature in °C.
• DCGM_FI_DEV_ECC_SBE_VOL_TOTAL: Single-bit ECC memory errors.

---

2. Deploying DCGM Exporter with Docker Compose

To monitor GPU devices, dcgm-exporter must run with GPU capabilities enabled and access to host device nodes.

Production docker-compose.yml Stack:

version: "3.8"

services:
  dcgm-exporter:
    image: nvcr.io/nvidia/k8s/dcgm-exporter:3.3.5-3.4.0-ubuntu22.04
    container_name: dcgm_exporter
    ports:
      - "9400:9400"
    environment:
      - DCGM_EXPORTER_LISTEN=:9400
      - DCGM_EXPORTER_INTERVAL=1000
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu, utility]
    restart: always

  prometheus:
    image: prom/prometheus:v2.49.0
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    restart: always

---

3. Configuring Prometheus Scrape Target

Add the DCGM Exporter target to your prometheus.yml configuration:

global:
  scrape_interval: 5s

scrape_configs:
  - job_name: "nvidia-dcgm"
    static_configs:
      - targets: ["dcgm-exporter:9400"]

Verify that Prometheus is scraping metrics successfully by opening http://<host-ip>:9090/targets.

---

4. Useful PromQL Queries for Alerting & Monitoring

Query 1: GPU Memory Utilization Percentage

Calculate VRAM usage percentage per GPU device:

(DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_FREE + DCGM_FI_DEV_FB_USED)) * 100

Query 2: Alert on Thermal Throttling

Detect GPUs exceeding 82°C:

DCGM_FI_DEV_GPU_TEMP > 82

Query 3: Total Power Consumption Across All GPUs

Sum power draw in Watts across the entire server:

sum(DCGM_FI_DEV_POWER_USAGE)

---

5. Grafana Dashboard Integration

Import official Grafana Dashboard ID 12239 (NVIDIA DCGM Exporter Dashboard) or construct custom panels using the PromQL metrics above.

---

Conclusion

NVIDIA DCGM Exporter provides production-grade GPU telemetry for containerized environments:

1. Deploy Container: Use nvcr.io/nvidia/k8s/dcgm-exporter with count: all GPU access.
2. Prometheus Scraping: Collect metrics at short intervals (5s–10s) for realtime inference monitoring.
3. Set Up Alerts: Create Prometheus rules for VRAM saturation (>95%) and high temperatures (>80°C).