GPU Multiplexing with NVIDIA MPS in Docker Containers

When deploying microservice architectures with multiple lightweight GPU inference containers (such as small BERT models, OCR services, or image feature extractors), running each container as an isolated CUDA application often leaves the physical GPU underutilized.

Standard CUDA execution enforces hardware context-switching between different processes. When multiple Docker containers attempt to submit CUDA kernels simultaneously, they suffer from high context-switch overhead, memory thrashing, and serialized execution.

NVIDIA CUDA MPS (Multi-Process Service) solves this by multiplexing multiple CUDA contexts into a single, shared GPU hardware context.

This technical guide explains how CUDA MPS works, how to start the host MPS control daemon, and how to configure Docker containers to connect to the shared MPS IPC pipe.

---

1. How CUDA MPS Works: Time-Slicing vs MPS

To understand MPS, compare the three ways of sharing a GPU among Docker containers:

1. Time-Slicing (Standard CUDA):
   [ Container A ] ───► [ Context Switch ] ───► [ Container B ] (High Latency Spike)

2. NVIDIA MPS (Multi-Process Service):
   [ Container A ] ──┐
   [ Container B ] ──┼──► [ MPS Control Daemon ] ──► [ Single GPU Context ] (Parallel Execution)
   [ Container C ] ──┘

Core Advantages of CUDA MPS:

• Overlapped Kernel Execution: Allows kernels from different Docker containers to run concurrently on available Streaming Multiprocessors (SMs).
• Reduced Memory Overhead: Eliminates per-process CUDA context allocation overhead (saving ~200MB–400MB of VRAM per container).
• Lower Latency: Replaces heavy hardware context-switching with lightweight IPC queue submissions.

---

2. Setting Up the Host MPS Daemon

The MPS control daemon (nvidia-cuda-mps-control) must run on the host system before launching Docker containers.

Step 1: Configure Environment Variables

Define the IPC pipes and logging directories on the host:

export CUDA_MPS_PIPE_DIRECTORY=/tmp/nvidia-mps
export CUDA_MPS_LOG_DIRECTORY=/tmp/nvidia-log

# Create directories with appropriate permissions
sudo mkdir -p $CUDA_MPS_PIPE_DIRECTORY $CUDA_MPS_LOG_DIRECTORY
sudo chmod 777 $CUDA_MPS_PIPE_DIRECTORY $CUDA_MPS_LOG_DIRECTORY

Step 2: Set Compute Mode to EXCLUSIVE_PROCESS

MPS requires the GPU compute mode to be set to EXCLUSIVE_PROCESS so that only the MPS control daemon can bind directly to the GPU:

sudo nvidia-smi -i 0 -c EXCLUSIVE_PROCESS

Step 3: Start the MPS Daemon

sudo nvidia-cuda-mps-control -d

Verify that the daemon is active:

ps aux | grep mps

---

3. Configuring Docker Containers to Access MPS

To allow Docker containers to submit CUDA jobs to the host MPS daemon, mount the host’s IPC pipe directory (/tmp/nvidia-mps) and set the matching environment variable inside the container.

Docker CLI Execution Command

docker run --rm -it \
  --device nvidia.com/gpu=all \
  -v /tmp/nvidia-mps:/tmp/nvidia-mps \
  -e CUDA_MPS_PIPE_DIRECTORY=/tmp/nvidia-mps \
  pytorch/pytorch:2.2.0-cuda12.1-cudnn8-runtime \
  python3 -c "import torch; print(torch.cuda.get_device_name(0))"

---

4. Docker Compose v2 Integration for MPS

Here is a docker-compose.yml deploying 3 parallel microservice containers sharing a single GPU via MPS:

version: "3.8"

services:
  ocr-service-1:
    image: custom-ocr-model:v1
    container_name: ocr_worker_1
    environment:
      - CUDA_MPS_PIPE_DIRECTORY=/tmp/nvidia-mps
    volumes:
      - /tmp/nvidia-mps:/tmp/nvidia-mps
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

  ocr-service-2:
    image: custom-ocr-model:v1
    container_name: ocr_worker_2
    environment:
      - CUDA_MPS_PIPE_DIRECTORY=/tmp/nvidia-mps
    volumes:
      - /tmp/nvidia-mps:/tmp/nvidia-mps
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

  embedding-service:
    image: custom-embedder:v1
    container_name: embedder_worker
    environment:
      - CUDA_MPS_PIPE_DIRECTORY=/tmp/nvidia-mps
    volumes:
      - /tmp/nvidia-mps:/tmp/nvidia-mps
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

---

5. Limiting VRAM and Compute Limits per Container under MPS

A common concern with MPS is a rogue container consuming 100% of the shared VRAM. MPS provides environment variables to enforce resource limits per container process.

Enforcing Memory Limits

Restrict a container to a maximum percentage of total VRAM (e.g., 25% of VRAM):

docker run --rm \
  --device nvidia.com/gpu=all \
  -v /tmp/nvidia-mps:/tmp/nvidia-mps \
  -e CUDA_MPS_PIPE_DIRECTORY=/tmp/nvidia-mps \
  -e CUDA_MPS_PINNED_STATIC_MEM_LIMIT=0=25G \
  custom-inference-app

Enforcing Thread/Active Execution Limits

Restrict a container to use at most 30% of the GPU’s Streaming Multiprocessors:

docker run --rm \
  --device nvidia.com/gpu=all \
  -v /tmp/nvidia-mps:/tmp/nvidia-mps \
  -e CUDA_MPS_PIPE_DIRECTORY=/tmp/nvidia-mps \
  -e CUDA_MPS_ACTIVE_THREAD_PERCENTAGE=30 \
  custom-inference-app

---

6. Stopping and Cleanup

To safely stop the host MPS daemon and return the GPU to default compute mode:

# Shutdown MPS Daemon
echo quit | sudo nvidia-cuda-mps-control

# Reset Compute Mode to DEFAULT
sudo nvidia-smi -i 0 -c DEFAULT

---

Conclusion

NVIDIA MPS multiplexing is the most efficient way to maximize throughput for multiple lightweight CUDA containers on a single GPU:

1. Start Host Daemon: Initialize nvidia-cuda-mps-control with EXCLUSIVE_PROCESS mode.
2. Mount Pipe Directory: Volume mount /tmp/nvidia-mps into all target Docker containers.
3. Enforce Hard Limits: Use CUDA_MPS_ACTIVE_THREAD_PERCENTAGE and CUDA_MPS_PINNED_STATIC_MEM_LIMIT to maintain multi-tenant fairness.