Migrating to NVIDIA CDI (Container Device Interface) with Docker Engine 25+

For years, accessing NVIDIA GPUs from Docker containers relied on architectural shims: first the nvidia-docker wrapper binary, and later the custom nvidia-container-runtime-hook intercepting container creation to inject host drivers and user-space CUDA libraries into the Linux namespace.

With the widespread adoption of cgroups v2, strict adherence to OCI (Open Container Initiative) specifications, and the release of Docker Engine 25+, this legacy approach is officially deprecated.

The industry-standard solution is CDI (Container Device Interface). Initiated in Containerd and hosted by the Cloud Native Computing Foundation (CNCF), the CDI specification allows complex hardware devices (GPUs, FPGAs, TPUs) to be declared via JSON specs without modifying the underlying container runtime.

This technical guide covers CDI architecture and provides a step-by-step procedure to migrate your Docker host to the new standard.

---

1. Limitations of the Legacy Model (nvidia-docker2 / Runtime Hook)

To understand the benefits of CDI, consider how the legacy architecture worked:

[ Docker Container ]
         │
         ▼
[ Docker Engine (dockerd) ]
         │
         ▼
[ nvidia-container-runtime ] ───► Intercepts OCI spec
         │
         ▼
[ nvidia-container-runtime-hook ] ───► Dynamically injects /dev/nvidia* & libcuda.so
         │
         ▼
[ Linux Kernel & Host Driver ]

Major Shortcomings of the Legacy Approach:

1. OCI Specification Violation: The runtime modified the OCI bundle after creation by Docker, causing unpredictable behavior across runc releases.
2. cgroups v2 Incompatibility: Managing device node permissions (/dev/nvidiaX) using cgroups v1 controllers does not map cleanly onto the unified cgroups v2 hierarchy.
3. Runtime Lock-in: Hosts were forced to replace the default runc runtime with nvidia in /etc/docker/daemon.json.

---

2. CDI Architecture: How It Works

CDI reverses this model: instead of intercepting container creation at runtime, the system generates a JSON specification document detailing the hardware devices available on the host machine.

[ CDI Spec File (/etc/cdi/nvidia.yaml) ]
         │
         ▼ (Declares: nvidia.com/gpu=0 -> /dev/nvidia0 + libs)
[ Docker Engine 25+ ] (Natively supports CDI spec)
         │
         ▼
[ Standard runc ] ───► Direct mount of declared devices without external hooks

A CDI device is identified using a Fully Qualified Device Name (FQDN) formatted as: vendor.com/device_type=device_id
Examples: nvidia.com/gpu=0 or nvidia.com/gpu=all.

---

3. Prerequisites and Installing the Updated NVIDIA Container Toolkit

System Requirements

• Linux Host (Ubuntu 22.04 LTS / 24.04 LTS or RHEL 9+)
• Proprietary NVIDIA Driver (version >= 535) installed on the host (nvidia-smi functional).
• Docker Engine 25.0 or higher (docker --version).

Package Installation

Remove legacy nvidia-docker2 packages if present:

sudo apt-get remove -y nvidia-docker2 nvidia-container-runtime

Install the modern NVIDIA Container Toolkit:

# Configure official NVIDIA repository
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb [arch=amd64] https://#deb [arch=amd64 signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

---

4. Generating the CDI Specification

Instead of configuring the Docker runtime, use nvidia-ctk to scan the host and write the CDI specification file to /etc/cdi:

sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

Inspecting the Generated Spec (/etc/cdi/nvidia.yaml)

cdiVersion: 0.5.0
kind: nvidia.com/gpu
devices:
  - name: "0"
    containerEdits:
      deviceNodes:
        - path: /dev/nvidia0
        - path: /dev/nvidiactl
        - path: /dev/nvidia-uvm
      mounts:
        - hostPath: /usr/lib/x86_64-linux-gnu/libcuda.so.535.129.03
          containerPath: /usr/lib/x86_64-linux-gnu/libcuda.so.1
          options:
            - ro
            - nosuid
            - nodev
            - bind
  - name: all
    containerEdits:
      deviceNodes:
        - path: /dev/nvidia0
        - path: /dev/nvidia1
        - path: /dev/nvidiactl
        - path: /dev/nvidia-uvm

---

5. Docker Engine Configuration (/etc/docker/daemon.json)

With CDI, you no longer need to add "default-runtime": "nvidia" to /etc/docker/daemon.json.

Keep your /etc/docker/daemon.json clean and minimal:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

Restart the Docker service to apply changes:

sudo systemctl restart docker

---

6. Validating GPU Passthrough via CDI

Docker Engine 25+ introduces the native --device flag compatible with CDI identifiers.

Test 1: Validate Access to All GPUs

docker run --rm --device nvidia.com/gpu=all ubuntu:22.04 nvidia-smi

Test 2: Assign a Specific GPU (GPU 0)

docker run --rm --device nvidia.com/gpu=0 ubuntu:22.04 nvidia-smi -L

Test 3: Docker Compose v2 Integration

With Docker Compose v2.24+, reference the CDI device directly inside the devices block:

version: "3.8"

services:
  cuda-test:
    image: nvidia/cuda:12.2.0-base-ubuntu22.04
    command: nvidia-smi
    devices:
      - nvidia.com/gpu=all

Launch the service:

docker compose up

---

7. Comparative Matrix: Legacy vs CDI

Feature	Legacy Model (nvidia-docker2)	CDI Model (Docker Engine 25+)
OCI Compliance	No (On-the-fly runtime modification)	Yes (Native OCI/CNCF specification)
Docker Runtime	Requires nvidia-container-runtime	Uses standard runc
cgroups v2 Support	Fragile / Workarounds required	Native
CLI Flag	--gpus all or --runtime=nvidia	--device nvidia.com/gpu=all
Multi-vendor	NVIDIA-only	Unified (NVIDIA, AMD ROCm, Intel)
Host Config	Heavy modifications to daemon.json	Isolated declarative spec in /etc/cdi/

---

Production Recommendations

1. Automate CDI Spec Generation: Run nvidia-ctk cdi generate on boot via systemd to update hardware devices dynamically.
2. Purge Legacy Configurations: Remove outdated runtime entries from /etc/docker/daemon.json.
3. Update CI/CD Pipelines: Replace deprecated --gpus flags with --device nvidia.com/gpu=all in integration tests.