Multi-GPU Orchestration with Docker Compose v2

In modern AI architectures, running multiple microservices backed by hardware accelerators (LLM inference engines like vLLM, vision models, GPU vector databases) requires precise device allocation on a single host.

Historically, Docker Compose relied on non-standard runtime options (runtime: nvidia). Under the Compose Specification v2, GPU resource allocation is fully standardized via the deploy.resources.reservations.devices block.

This guide provides practical patterns for orchestrating multi-GPU workloads using GPU counts, physical index targeting, and hardware UUID bindings.

---

1. The Compose v2 Specification for GPUs

The GPU allocation block in Docker Compose v2 uses the devices directive under reservations:

deploy:
  resources:
    reservations:
      devices:
        - driver: nvidia
          count: all # Or specific count: 1, 2, etc.
          capabilities: [gpu]

Core Configuration Options:

• driver: Set to nvidia (or omit when using CDI).
• count: Number of GPUs to allocate (1, 2, all).
• device_ids: Explicit list of GPU identifiers (index "0", "1" or hardware UUID "GPU-xxx").
• capabilities: Requested hardware feature sets ([gpu], [gpu, utility], [gpu, compute]).

---

2. Practical YAML Configuration Patterns

Pattern 1: Allocate All Available GPUs to a Single Service (Tensor Parallelism)

To deploy large models (e.g., Llama-3 70B via vLLM) requiring 4 GPUs with NCCL inter-GPU communication:

version: "3.8"

services:
  vllm-llama70b:
    image: vllm/vllm-openai:latest
    container_name: vllm_cluster
    ports:
      - "8000:8000"
    ipc: host
    shm_size: "16gb"
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    command: >
      --model meta-llama/Meta-Llama-3-70B-Instruct
      --tensor-parallel-size 4
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

---

Pattern 2: Explicit Isolation by Physical Index (GPU 0 vs GPU 1)

To pin independent microservices to dedicated physical GPUs:

version: "3.8"

services:
  ollama-gpu-0:
    image: ollama/ollama:latest
    container_name: service_ollama
    ports:
      - "11434:11434"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["0"]
              capabilities: [gpu]

  whisper-gpu-1:
    image: fedora/whisper-service:latest
    container_name: service_whisper
    ports:
      - "5000:5000"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["1"]
              capabilities: [gpu]

---

Pattern 3: Target by Hardware GPU UUID (Production Immutable Pinning)

On multi-card servers, PCIe enumeration order (0, 1, 2) can shift after kernel reboots. For immutable production deployments, target Hardware UUIDs retrieved via nvidia-smi -L:

version: "3.8"

services:
  vision-service:
    image: custom-vision-model:v1
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["GPU-a1b2c3d4-e5f6-7890-1234-56789abcdef0"]
              capabilities: [gpu]

---

3. Environment Variables and Common Gotchas

The CUDA_VISIBLE_DEVICES Behavior

When Docker Compose passes reserved GPUs into a container, CUDA re-indexes visible GPUs starting at 0.

Example: If you assign device_ids: ["2", "3"] to a container:

• Inside the container, CUDA_VISIBLE_DEVICES will be set to 0,1 (mapping to real host GPUs 2 and 3).
• PyTorch/CUDA scripts inside the container should target cuda:0 and cuda:1.

Avoid Mixing runtime: nvidia and deploy.resources

# ❌ INCORRECT (Do not mix runtime with deploy.resources)
services:
  app:
    image: my-app
    runtime: nvidia # ❌ Remove this line
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

---

4. Verifying Multi-GPU Topology

Verify that P2P / NVLink interconnects are functional between assigned GPUs by running nvidia-smi topo -m inside the running container:

docker compose exec vllm_cluster nvidia-smi topo -m

Sample output confirming NVLink matrix:

        GPU0    GPU1    GPU2    GPU3
GPU0     X      NV4     NV4     NV4
GPU1    NV4      X      NV4     NV4
GPU2    NV4     NV4      X      NV4
GPU3    NV4     NV4     NV4      X

---

Conclusion

Multi-GPU orchestration with Docker Compose v2 yields clean, reproducible Infrastructure-as-Code setups:

1. Use count: all for distributed Tensor Parallelism models.
2. Use device_ids with hardware UUIDs to isolate microservices across host reboots.
3. Include shm_size: "16gb" on all multi-GPU Compose services to prevent inter-process IPC bottlenecks.

Heterogeneous GPU Device Isolation & CUDA_VISIBLE_DEVICES

When managing servers with mixed GPU generations (e.g. an RTX 4090 paired with an RTX 3090), you must explicitly isolate specific device IDs to individual Docker container services.

Docker Compose Explicit Device Assignment

version: "3.8"

services:
  heavy_llm_service:
    image: vllm/vllm-openai:latest
    container_name: llm_gpu_0
    environment:
      - CUDA_VISIBLE_DEVICES=0
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]

  vision_model_service:
    image: ollama/ollama:latest
    container_name: vision_gpu_1
    environment:
      - CUDA_VISIBLE_DEVICES=1
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['1']
              capabilities: [gpu]

Multi-GPU Orchestration Topology

Orchestrating multi-GPU LLM serving with Docker Compose requires fine-grained control over GPU device mapping, NVLink peer-to-peer memory access, and NUMA node affinity.

Improper device assignment leads to PCIe bandwidth bottlenecks when data traverses across CPU sockets instead of using direct GPU-to-GPU NVLink bridges.

+-------------------------------------------------------------------------+
|                           Physical Host System                          |
|  +-----------------------------------+   +---------------------------+  |
|  | NUMA Node 0 (CPUs 0-31)           |   | NUMA Node 1 (CPUs 32-63)  |  |
|  | GPUs: /dev/nvidia0, /dev/nvidia1  |   | GPUs: /dev/nvidia2, 3     |  |
|  +-----------------------------------+   +---------------------------+  |
|                   || NVLink Peer-to-Peer Interconnect                    |
+-------------------||-----------------------------------------------------+
                    \/
+-------------------------------------------------------------------------+
|                  Docker Compose Container Ecosystem                     |
|  +------------------------------+     +------------------------------+  |
|  | Container 1: High-Priority   |     | Container 2: Background      |  |
|  | Model (GPUs 0, 1)            |     | Fine-Tuning (GPUs 2, 3)      |  |
|  +------------------------------+     +------------------------------+  |
+-------------------------------------------------------------------------+

Advanced Multi-Service docker-compose.yml

This manifest demonstrates orchestrating two separate isolated GPU workloads—an vLLM production inference engine on GPUs 0 & 1, and an Unsloth fine-tuning pipeline on GPUs 2 & 3:

version: "3.8"

services:
  production-vllm:
    image: vllm/vllm-openai:v0.6.2
    container_name: vllm-inference-prod
    restart: always
    environment:
      - CUDA_VISIBLE_DEVICES=0,1
      - NCCL_P2P_DISABLE=0
      - NCCL_IB_DISABLE=1
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0', '1']
              capabilities: [gpu]
    ipc: host
    ports:
      - "8000:8000"
    command: >
      --model mistralai/Mixtral-8x7B-Instruct-v0.1
      --tensor-parallel-size 2
      --gpu-memory-utilization 0.90

  train-worker:
    image: unsloth/unsloth:latest
    container_name: fine-tune-job
    environment:
      - CUDA_VISIBLE_DEVICES=2,3
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['2', '3']
              capabilities: [gpu]
    ipc: host
    volumes:
      - ./datasets:/workspace/datasets
      - ./checkpoints:/workspace/checkpoints
    command: python3 /workspace/train.py

Performance Verification & Topo Inspection

Verify GPU topology mapping before running multi-GPU jobs:

# Check NVLink connection Matrix
nvidia-smi topo -m

# Output expected matrix:
#        GPU0  GPU1  GPU2  GPU3
# GPU0   X     NV4   SYS   SYS
# GPU1   NV4   X     SYS   SYS
# GPU2   SYS   SYS   X     NV4
# GPU3   SYS   SYS   NV4   X