AMD ROCm GPU Passthrough under Docker with Linux cgroups v2

While NVIDIA CUDA has dominated containerized AI workloads, AMD’s ROCm (Radeon Open Compute) ecosystem has emerged as a powerful open-source alternative for high-performance machine learning inference and training on AMD Instinct (MI200/MI300 series) and Radeon GPUs.

Unlike NVIDIA’s Container Toolkit which uses custom container runtime shims, AMD ROCm relies on standard Linux kernel device nodes (/dev/kfd and /dev/dri) mapped directly into containers via Linux cgroups v2 permissions.

This technical guide covers the architecture of AMD ROCm GPU passthrough, host kernel dependencies, Docker CLI/Compose configurations, and user group permission handling.

---

1. Architecture of AMD ROCm GPU Passthrough

AMD’s GPU compute architecture relies on two critical device driver interfaces in the Linux kernel:

[ Docker Container ]
         │
         ├──► /dev/kfd  (Kernel Fusion Driver - Compute & Memory Management)
         └──► /dev/dri  (Direct Rendering Infrastructure - Display & Memory Rendering)
         │
         ▼
[ Linux Kernel Subsystem (amdgpu & kfd) ]
         │
         ▼
[ AMD Hardware (Instinct MI300X / Radeon RX 7900 XTX) ]

1. /dev/kfd (Kernel Fusion Driver): Manages user-space queue submissions, memory allocations, and inter-process communication for compute tasks.
2. /dev/dri (Direct Rendering Infrastructure): Provides direct access to GPU render nodes (/dev/dri/renderD128, etc.) for memory allocation and hardware acceleration.

---

2. Host System Prerequisites

Step 1: Verify Host Kernel Driver

Ensure the amdgpu kernel module and KFD driver are loaded on the host machine:

lsmod | grep amdgpu

Check that both /dev/kfd and /dev/dri device nodes exist:

ls -l /dev/kfd /dev/dri

Step 2: User Group Permissions

On modern Linux distributions, access to /dev/dri and /dev/kfd is restricted to the video and render system groups.

Identify the Group IDs (GIDs) on your host:

getent group video | cut -d: -f3
getent group render | cut -d: -f3

Example Output: video: 44, render: 109.

---

3. Docker CLI Configuration for AMD ROCm

To pass AMD GPUs into a Docker container without grant full --privileged access, mount /dev/kfd and /dev/dri while mapping the host system groups.

Standard Docker CLI Command

docker run --rm -it \
  --device=/dev/kfd \
  --device=/dev/dri \
  --group-add render \
  --group-add video \
  --ipc=host \
  rocm/pytorch:rocm6.0_ubuntu22.04_py3.10_pytorch_2.1.1 \
  rocm-smi

Explanation of Flags:

• --device=/dev/kfd: Exposes the kernel compute interface required for ROCm memory allocation.
• --device=/dev/dri: Exposes all Direct Rendering render nodes.
• --group-add render: Grants the container process membership in the host’s render group to bypass permission denied errors.
• --ipc=host: Ensures shared memory availability for PyTorch worker processes.

---

4. Docker Compose v2 Configuration

Here is a production-ready docker-compose.yml for serving an LLM using PyTorch ROCm:

version: "3.8"

services:
  rocm-llm-service:
    image: rocm/pytorch:rocm6.0_ubuntu22.04_py3.10_pytorch_2.1.1
    container_name: rocm_pytorch_server
    ports:
      - "8000:8000"
    ipc: host
    shm_size: "16gb"
    devices:
      - "/dev/kfd:/dev/kfd"
      - "/dev/dri:/dev/dri"
    group_add:
      - "render"
      - "video"
    security_opt:
      - seccomp:unconfined
    environment:
      - HIP_VISIBLE_DEVICES=0
    volumes:
      - ./models:/workspace/models
    command: python3 -m torch.utils.collect_env

---

5. Verifying ROCm Acceleration in the Container

Once inside the container, run AMD’s system management interface tool (rocm-smi) and verify PyTorch ROCm detection:

Verification Command 1: rocm-smi

rocm-smi

Expected output showing AMD GPU status:

========================= ROCm System Management Interface =========================
====================================================================================
GPU  Temp   AvgPwr  SCLK    MCLK    VRAM%  GPU%  HIP Device ID
0    34.0C  28.0W   800Mhz  960Mhz   0%    0%    0x744c
====================================================================================

Verification Command 2: PyTorch HIP Detection

python3 -c "import torch; print('HIP Available:', torch.cuda.is_available()); print('Device Name:', torch.cuda.get_device_name(0))"

Output:

HIP Available: True
Device Name: AMD Radeon RX 7900 XTX

(Note: PyTorch on ROCm uses the torch.cuda API namespace via HIP translation).

---

6. Troubleshooting Common Issues

Issue 1: Permission denied: '/dev/kfd'

Cause: The user inside the container does not belong to the host render or video GID.
Fix: Pass exact numeric GIDs via --group-add:

docker run --group-add $(getent group render | cut -d: -f3) ...

Issue 2: HIP Error: No binary compatible device found

Cause: Your consumer GPU architecture (GFX version) is not officially targeted by the pre-compiled ROCm binaries.
Fix: Override the target GFX architecture via environment variable before launching Python:

export HSA_OVERRIDE_GFX_VERSION=11.0.0 # For RX 7000 Series (RDNA3)

---

Conclusion

AMD ROCm GPU passthrough in Docker offers a clean, native approach without requiring custom container runtimes:

1. Map both /dev/kfd and /dev/dri device nodes into the container.
2. Add render and video group memberships via --group-add.
3. Set HSA_OVERRIDE_GFX_VERSION for consumer RDNA3 GPUs when necessary.