AMD ROCm GPU Passthrough under Docker with Linux cgroups v2

Overview

While NVIDIA CUDA dominates GPU AI deployments, AMD’s ROCm (Radeon Open Compute) framework provides a powerful open-source alternative for running LLMs on AMD Radeon (RX 7900 XTX) and Instinct (MI250/MI300) hardware. Exposing AMD GPUs to Docker containers requires passing kernel character devices (/dev/kfd and /dev/dri) with render group permissions under cgroups v2.

This guide provides the complete setup for configuring ROCm 6.x GPU acceleration in Docker.

---

AMD ROCm Device Access Architecture

┌────────────────────────────────────────────────────────┐
│                   LINUX KERNEL (6.x)                   │
│  - amdgpu Kernel Module                                │
│  - Kernel Fusion Driver (/dev/kfd)                     │
│  - Direct Rendering Manager (/dev/dri/card* & render*) │
└──────────────┬─────────────────────────────────────────┘
               │ Host Permissions: video, render groups
               ▼
┌────────────────────────────────────────────────────────┐
│                   DOCKER CONTAINER                     │
│  --device=/dev/kfd --device=/dev/dri                   │
│  --group-add render                                    │
│  ROCm PyTorch Engine (rocm6.1_ubuntu22.04)             │
└────────────────────────────────────────────────────────┘

---

Step 1: Add Host User Permissions

Add the Docker host user to the video and render system groups:

sudo usermod -aG video $USER
sudo usermod -aG render $USER
newgrp render

Verify character device node permissions:

ls -la /dev/kfd /dev/dri

Expected Output: crw-rw---- 1 root render /dev/kfd

---

Step 2: Production Docker Compose Configuration

Create /opt/rocm-stack/docker-compose.yml:

version: "3.8"

services:
  rocm_ollama:
    image: ollama/ollama:rocm
    container_name: rocm_ollama_engine
    restart: unless-stopped
    devices:
      - "/dev/kfd:/dev/kfd"
      - "/dev/dri:/dev/dri"
    group_add:
      - "109" # Render group ID on host
      - "44"  # Video group ID on host
    ipc: host
    cap_add:
      - SYS_PTRACE
    security_opt:
      - seccomp=unconfined
    environment:
      - HSA_OVERRIDE_GFX_VERSION=11.0.0 # Required for RX 7900 XTX
      - ROCM_PATH=/opt/rocm
    ports:
      - "11434:11434"
    volumes:
      - rocm_storage:/root/.ollama

volumes:
  rocm_storage:

---

Step 3: Verification & PyTorch ROCm Smoke Test

Execute a PyTorch CUDA-compatibility check inside the container:

docker run --rm   --device=/dev/kfd --device=/dev/dri   --group-add render   rocm/pytorch:rocm6.1_ubuntu22.04   python3 -c "import torch; print('ROCm Available:', torch.cuda.is_available()); print('Device Name:', torch.cuda.get_device_name(0))"

Expected Successful Output:

ROCm Available: True
Device Name: AMD Radeon RX 7900 XTX

---

Diagnostic Matrix

Error Message	Root Cause	Solution
hipErrorNoBinaryForGpu	GFX target architecture mismatch	Set HSA_OVERRIDE_GFX_VERSION=11.0.0 for RDNA3 or 10.3.0 for RDNA2.
Permission denied: /dev/kfd	Container missing render group	Add group_add: ["render"] or host GID to Compose file.
rocm-smi command not found	ROCm user tools missing	Mount /opt/rocm binaries into container environment.

Custom PyTorch ROCm Dockerfile & Telemetry Setup

To achieve peak inference speeds on AMD Radeon RX 7900 XTX or Instinct MI300 hardware, build a custom optimized Docker container using the official AMD ROCm base image.

Optimized Dockerfile (Dockerfile.rocm)

FROM rocm/pytorch:rocm6.1_ubuntu22.04

# Enable High-Performance Flash Attention for ROCm
ENV HSA_OVERRIDE_GFX_VERSION=11.0.0
ENV ROCM_PATH=/opt/rocm
ENV PYTHONUNBUFFERED=1

WORKDIR /workspace

# Install ROCm monitoring utilities
RUN apt-get update && apt-get install -y rocm-smi-lib

EXPOSE 8000
CMD ["python3", "-m", "vllm.entrypoints.openai.api_server", "--host", "0.0.0.0", "--port", "8000"]

Real-Time AMD GPU Telemetry

Monitor VRAM allocations, power consumption, and GPU clock speeds using rocm-smi:

docker exec -it rocm_ollama_engine rocm-smi --showuse --showtemp --showpower

AMD ROCm Container Architecture Deep Dive

Deploying enterprise AI workloads on AMD Instinct GPUs (MI210, MI250, MI300X) or Radeon RX GPUs requires presenting AMD’s kernel driver stack—specifically /dev/kfd (Kernel Fusion Driver) and /dev/dri (Direct Rendering Infrastructure)—into LXC containers or Docker runtimes.

Unlike NVIDIA’s dynamic CUDA injection hook (nvidia-container-toolkit), AMD’s ROCm runtime relies on direct POSIX device node passthrough alongside user-space ROCm libraries (HIP, ROCm-SMI, rocBLAS) built inside target container images.

+-----------------------------------------------------------------------+
|                    ROCm PyTorch Docker Container                      |
|  +-----------------------------------------------------------------+  |
|  | HIP / PyTorch / rocBLAS App Runtime                             |  |
|  +-----------------------------------------------------------------+  |
+-----------------------------------|-----------------------------------+
                                    | /dev/kfd & /dev/dri Direct Access
+-----------------------------------|-----------------------------------+
|                        Host Linux Kernel (ROCm)                       |
|  +------------------------+             +--------------------------+  |
|  | amdkfd Kernel Driver   |             | amdgpu DRM Kernel Driver |  |
|  +------------------------+             +--------------------------+  |
+-----------------------------------------------------------------------+

Advanced Multi-GPU vLLM Compose Configuration

Create a robust docker-compose.yml for serving LLMs via ROCm vLLM or Ollama containers:

version: "3.8"

services:
  rocm-vllm:
    image: rocm/vllm:rocm6.1_mi300_ubuntu22.04_py3.10_vllm_0.6.0
    container_name: rocm-vllm-engine
    restart: unless-stopped
    devices:
      - "/dev/kfd:/dev/kfd"
      - "/dev/dri:/dev/dri"
    group_add:
      - "video"
      - "109" # Host 'render' group ID (verify via getent group render)
    security_opt:
      - seccomp:unconfined
    cap_add:
      - SYS_PTRACE
    ipc: host
    environment:
      - ROCM_PATH=/opt/rocm
      - HIP_VISIBLE_DEVICES=0,1,2,3
      - HSA_OVERRIDE_GFX_VERSION=11.0.0 # Force RDNA3 fallback if required
      - PYTORCH_ROCM_ALLOC_CONF=garbage_collection_threshold:0.8,max_split_size_mb:512
    ports:
      - "8000:8000"
    volumes:
      - ./models:/root/.cache/huggingface
    command: >
      --model Qwen/Qwen2.5-72B-Instruct-AWQ
      --tensor-parallel-size 4
      --gpu-memory-utilization 0.92
      --max-model-len 8192

Validation & HIP Device Discovery

Execute interactive diagnostic checks to confirm HIP GPU memory allocation:

# Execute rocm-smi inside container
docker exec -it rocm-vllm-engine rocm-smi

# Verify HIP PyTorch CUDA translation layer
docker exec -it rocm-vllm-engine python3 -c "
import torch
print('CUDA Available via ROCm:', torch.cuda.is_available())
print('Device Count:', torch.cuda.device_count())
print('Device Name:', torch.cuda.get_device_name(0))
"