NVIDIA GPU Monitoring with DCGM Exporter & Prometheus in Docker

Overview

Monitoring GPU metrics—such as SM (Streaming Multiprocessor) utilization, NVLink bandwidth, PCIe throughput, temperature, and memory allocation—is essential for running high-availability AI inference infrastructure.

Standard host metrics (via Node Exporter) do not capture granular GPU telemetry. NVIDIA Data Center GPU Manager (DCGM) Exporter exposes low-overhead Prometheus metrics directly from the kernel driver.

This guide details a production Docker deployment for collecting GPU telemetry via DCGM Exporter, scraping metrics with Prometheus, and visualizing them in Grafana.

---

System Architecture

┌────────────────────────────────────────────────────────┐
│                   GPU INFRASTRUCTURE HOST              │
│                                                        │
│  ┌────────────────────────┐                            │
│  │ NVIDIA Kernel Driver   │                            │
│  └───────────┬────────────┘                            │
│              │ Low-Level NVML / DCGM Engine            │
│              ▼                                         │
│  ┌────────────────────────┐                            │
│  │ DCGM Exporter Container│ ──► Port 9400 (/metrics)   │
│  └───────────┬────────────┘                            │
└──────────────┼─────────────────────────────────────────┘
               │ Prometheus Scrape (15s Interval)
               ▼
┌────────────────────────────────────────────────────────┐
│              PROMETHEUS & GRAFANA STACK                │
│  - Prometheus Container (Port 9090)                    │
│  - Grafana Dashboard Container (Port 3000)             │
└────────────────────────────────────────────────────────┘

---

Step 1: Deploy DCGM Exporter in Docker Compose

Create /opt/gpu-monitoring/docker-compose.yml:

version: "3.8"

services:
  dcgm-exporter:
    image: nvcr.io/nvidia/k8s/dcgm-exporter:3.3.5-3.4.0-ubuntu22.04
    container_name: dcgm_exporter
    restart: unless-stopped
    environment:
      - DCGM_EXPORTER_INTERVAL=1000
    ports:
      - "9400:9400"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu, compute, utility]
    cap_add:
      - SYS_ADMIN
    networks:
      - monitoring

  prometheus:
    image: prom/prometheus:v2.52.0
    container_name: prometheus
    restart: unless-stopped
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prom_data:/prometheus
    ports:
      - "9090:9090"
    networks:
      - monitoring

  grafana:
    image: grafana/grafana:11.0.0
    container_name: grafana
    restart: unless-stopped
    ports:
      - "3000:3000"
    volumes:
      - grafana_data:/var/lib/grafana
    networks:
      - monitoring

networks:
  monitoring:
    driver: bridge

volumes:
  prom_data:
  grafana_data:

---

Step 2: Configure Prometheus Scrape Job

Create /opt/gpu-monitoring/prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'nvidia_dcgm'
    static_configs:
      - targets: ['dcgm-exporter:9400']
        labels:
          environment: 'production-gpu'
          cluster: 'sovereign-ai-node-01'

---

Step 3: Essential Prometheus Alerts for GPU Stability

Add alert rules to /opt/gpu-monitoring/prometheus.yml to trigger warnings before hardware thermal throttling or VRAM exhaustion occurs:

rule_files:
  - "gpu_alerts.yml"

Create /opt/gpu-monitoring/gpu_alerts.yml:

groups:
  - name: GPUHardwareAlerts
    rules:
      - alert: GPUTemperatureHigh
        expr: DCGM_FI_DEV_GPU_TEMP > 82
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "GPU Temperature High on {{ $labels.gpu }}"
          description: "GPU thermal sensors report {{ $value }}°C (Threshold: 82°C)."

      - alert: GPUVRAMExhausted
        expr: (DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE)) * 100 > 95
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "GPU Memory Near Limit on {{ $labels.gpu }}"
          description: "VRAM utilization is at {{ $value }}%."

---

Metric Reference Table

Metric Name	Description	Key Alert Threshold
DCGM_FI_DEV_GPU_UTIL	Streaming Multiprocessor (SM) utilization %	Near 100% during batch inference
DCGM_FI_DEV_MEM_COPY_UTIL	Memory controller activity %	Identifies memory bandwidth bottlenecks
DCGM_FI_DEV_GPU_TEMP	Core GPU die temperature (°C)	Warning > 80°C, Throttle > 87°C
DCGM_FI_DEV_POWER_USAGE	Real-time power draw in Watts	Compare against Board TDP Limit
DCGM_FI_DEV_FB_FREE	Free Framebuffer VRAM (MB)	Critical < 1000 MB

---

Verification & Smoke Test

1. Verify DCGM metrics endpoint:

curl http://localhost:9400/metrics | grep DCGM_FI_DEV_GPU_TEMP

2. Confirm Prometheus target status: Navigate to http://localhost:9090/targets and verify dcgm-exporter status is UP.

---

Summary Checklist

• Deployed DCGM exporter container with NVIDIA runtime access.
• Configured Prometheus scraping job on port 9400.
• Defined thermal and memory threshold alert rules.

1. Host Preparation & Kernel Optimization

Configure host kernel parameters to support high-throughput network sockets and memory allocation for containerized workloads. Edit /etc/sysctl.d/99-sovereign-ai.conf:

# Sovereign AI Host Tuning Parameters
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535
fs.file-max = 2097152
vm.max_map_count = 262144

Apply sysctl changes:

sudo sysctl --system

2. Production Docker Service Specification

Deploy the container stack using hardened runtime security parameters, non-root user mappings, and explicit resource allocations:

version: "3.8"

services:
  core_workload:
    image: opsnexus/engine:v1.4
    container_name: sovereign_core_workload
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    cap_drop:
      - ALL
    cap_add:
      - SYS_ADMIN
    environment:
      - LOG_LEVEL=info
      - METRICS_ENABLED=true
    volumes:
      - /opt/data:/app/data:rw
    ports:
      - "127.0.0.1:8080:8080"
    deploy:
      resources:
        limits:
          cpus: '8.0'
          memory: 32G
        reservations:
          cpus: '4.0'
          memory: 16G

---

Failure Mode 1: High VRAM Fragmentation under Continuous Batching

During extended execution, dynamic memory allocation can create fragmented VRAM memory blocks, causing unexpected out-of-memory errors even when reported total free memory appears sufficient.

• Diagnostic Command: Run nvidia-smi --query-gpu=memory.used,memory.free --format=csv every second during peak request surges.
• Resolution: Enable unified memory allocators or set PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True".

Failure Mode 2: Container Bridge Network Throughput Degradation

Heavy payload transfers between containerized proxy sidecars and backend engines can saturate default Docker bridge network buffers.

• Diagnostic Command: Test inter-container bandwidth using iperf3 between connected container sockets.
• Resolution: Tune network socket buffer sizes or utilize host networking mode (network_mode: "host") for low-latency internal traffic.

---

GPU Telemetry & Monitoring Architecture

Enterprise AI clusters require real-time visibility into GPU utilization, VRAM usage, NVLink error rates, memory temperature, and power throttling.

The NVIDIA Data Center GPU Manager (DCGM) Exporter pulls low-level hardware metrics directly from NVML and exposes a Prometheus endpoint on port 9400.

+-------------------------------------------------------------------------+
|                 NVIDIA DCGM Monitoring Infrastructure                   |
|  +-------------------------------------------------------------------+  |
|  | Host GPU (NVML Interface) -> DCGM Exporter Container (Port 9400)  |  |
|  +-------------------------------------------------------------------+  |
|                                   || Scrape Metrics                     |
|                                   \/                                    |
|  +-------------------------------------------------------------------+  |
|  | Prometheus Time-Series DB -> Grafana Monitoring Dashboards        |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

Production Docker Compose Monitoring Stack (docker-compose.yml)

version: "3.8"

services:
  dcgm-exporter:
    image: nvcr.io/nvidia/k8s/dcgm-exporter:3.3.5-3.4.0-ubuntu22.04
    container_name: dcgm-exporter
    restart: always
    runtime: nvidia
    environment:
      - DCGM_EXPORTER_LISTEN_ADDRESS=:9400
    ports:
      - "9400:9400"
    cap_add:
      - SYS_ADMIN

  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: always
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml