Multi-GPU Orchestration with Docker Compose v2

In modern AI architectures, running multiple microservices backed by hardware accelerators (LLM inference engines like vLLM, vision models, GPU vector databases) requires precise device allocation on a single host.

Historically, Docker Compose relied on non-standard runtime options (runtime: nvidia). Under the Compose Specification v2, GPU resource allocation is fully standardized via the deploy.resources.reservations.devices block.

This guide provides practical patterns for orchestrating multi-GPU workloads using GPU counts, physical index targeting, and hardware UUID bindings.

---

1. The Compose v2 Specification for GPUs

The GPU allocation block in Docker Compose v2 uses the devices directive under reservations:

deploy:
  resources:
    reservations:
      devices:
        - driver: nvidia
          count: all # Or specific count: 1, 2, etc.
          capabilities: [gpu]

Core Configuration Options:

• driver: Set to nvidia (or omit when using CDI).
• count: Number of GPUs to allocate (1, 2, all).
• device_ids: Explicit list of GPU identifiers (index "0", "1" or hardware UUID "GPU-xxx").
• capabilities: Requested hardware feature sets ([gpu], [gpu, utility], [gpu, compute]).

---

2. Practical YAML Configuration Patterns

Pattern 1: Allocate All Available GPUs to a Single Service (Tensor Parallelism)

To deploy large models (e.g., Llama-3 70B via vLLM) requiring 4 GPUs with NCCL inter-GPU communication:

version: "3.8"

services:
  vllm-llama70b:
    image: vllm/vllm-openai:latest
    container_name: vllm_cluster
    ports:
      - "8000:8000"
    ipc: host
    shm_size: "16gb"
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    command: >
      --model meta-llama/Meta-Llama-3-70B-Instruct
      --tensor-parallel-size 4
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

---

Pattern 2: Explicit Isolation by Physical Index (GPU 0 vs GPU 1)

To pin independent microservices to dedicated physical GPUs:

version: "3.8"

services:
  ollama-gpu-0:
    image: ollama/ollama:latest
    container_name: service_ollama
    ports:
      - "11434:11434"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["0"]
              capabilities: [gpu]

  whisper-gpu-1:
    image: fedora/whisper-service:latest
    container_name: service_whisper
    ports:
      - "5000:5000"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["1"]
              capabilities: [gpu]

---

Pattern 3: Target by Hardware GPU UUID (Production Immutable Pinning)

On multi-card servers, PCIe enumeration order (0, 1, 2) can shift after kernel reboots. For immutable production deployments, target Hardware UUIDs retrieved via nvidia-smi -L:

version: "3.8"

services:
  vision-service:
    image: custom-vision-model:v1
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["GPU-a1b2c3d4-e5f6-7890-1234-56789abcdef0"]
              capabilities: [gpu]

---

3. Environment Variables and Common Gotchas

The CUDA_VISIBLE_DEVICES Behavior

When Docker Compose passes reserved GPUs into a container, CUDA re-indexes visible GPUs starting at 0.

Example: If you assign device_ids: ["2", "3"] to a container:

• Inside the container, CUDA_VISIBLE_DEVICES will be set to 0,1 (mapping to real host GPUs 2 and 3).
• PyTorch/CUDA scripts inside the container should target cuda:0 and cuda:1.

Avoid Mixing runtime: nvidia and deploy.resources

# ❌ INCORRECT (Do not mix runtime with deploy.resources)
services:
  app:
    image: my-app
    runtime: nvidia # ❌ Remove this line
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

---

4. Verifying Multi-GPU Topology

Verify that P2P / NVLink interconnects are functional between assigned GPUs by running nvidia-smi topo -m inside the running container:

docker compose exec vllm_cluster nvidia-smi topo -m

Sample output confirming NVLink matrix:

        GPU0    GPU1    GPU2    GPU3
GPU0     X      NV4     NV4     NV4
GPU1    NV4      X      NV4     NV4
GPU2    NV4     NV4      X      NV4
GPU3    NV4     NV4     NV4      X

---

Conclusion

Multi-GPU orchestration with Docker Compose v2 yields clean, reproducible Infrastructure-as-Code setups:

1. Use count: all for distributed Tensor Parallelism models.
2. Use device_ids with hardware UUIDs to isolate microservices across host reboots.
3. Include shm_size: "16gb" on all multi-GPU Compose services to prevent inter-process IPC bottlenecks.