Multi-Tenant Container GPU Security and Hardening

In multi-tenant cloud environments or shared enterprise GPU clusters, granting Docker containers access to host GPU hardware introduces significant security surface area.

Because GPU containers share the host Linux kernel’s CUDA driver modules (nvidia.ko, nvidia-uvm.ko), insecure container configurations can lead to cross-container VRAM data leakage, privilege escalation to host root, or Denial-of-Service (DoS) attacks via memory exhaustion.

This technical guide outlines threat vectors unique to containerized GPUs and provides a hardening blueprint to secure multi-tenant GPU workloads.

---

1. Primary GPU Security Attack Vectors in Containers

[ Malicious Container ]
         │
         ├── Vector 1: Uncleared VRAM Memory Residuals (Data Leakage)
         ├── Vector 2: Direct Driver Exploitation via /dev/nvidiactl
         └── Vector 3: DoS via Host GPU Memory Saturation
         │
         ▼
[ Shared Host NVIDIA Kernel Module (nvidia.ko) ]

1. Uncleared Framebuffer (VRAM) Residuals: When a CUDA process terminates, residual data (e.g., model weights, sensitive prompt tokens) may linger in uninitialized GPU VRAM allocations accessed by a subsequent container.
2. Container Breakout via Driver Vulnerabilities: Granting --privileged mode or full host IPC access to GPU containers allows malicious code to interact directly with host kernel modules.
3. Denial of Service (GPU Memory Exhaustion): A single tenant allocating 100% of physical VRAM can crash or stall co-located inference containers.

---

2. Hardening Action Plan

Rule 1: Never Use --privileged with GPU Containers

Passing --privileged to a container with GPU access disables Linux seccomp profiles, AppArmor/SELinux protections, and capability restrictions.

# ❌ INSECURE: Do not run GPU containers with --privileged
docker run --privileged --gpus all my-gpu-app

# ✅ SECURE: Pass explicit CDI device references with default security isolation
docker run --device nvidia.com/gpu=all my-gpu-app

---

Rule 2: Enforce Environment Isolation with NVIDIA_REQUIRE_*

Use the NVIDIA_REQUIRE_* environment variables to enforce driver capability constraints. For example, prevent containers from calling host management APIs (nvidia-smi or driver management functions):

version: "3.8"

services:
  secure-inference:
    image: my-tenant-app:v1
    environment:
      # Restrict driver access to CUDA compute workloads only (block management APIs)
      - NVIDIA_DRIVER_CAPABILITIES=compute,utility
      # Enforce CUDA version constraints
      - NVIDIA_REQUIRE_CUDA=cuda>=12.0
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

---

Rule 3: Enable Deterministic VRAM Memory Sanitization

Ensure the host driver zeros out GPU VRAM allocations upon process exit to prevent cross-tenant memory inspection:

Check current driver memory scrubbing status on the host:

cat /sys/module/nvidia/parameters/NVreg_EnableUserMEMMap

Enforce memory sanitization by configuring host module parameters:

echo "options nvidia NVreg_InitializeSystemMemoryAllocations=1" | sudo tee /etc/modprobe.d/nvidia-security.conf

---

Rule 4: Enforce Per-Tenant GPU Allocation (MIG or Memory Limits)

To prevent a single tenant from starving other containers:

1. Hardware Isolation (Preferred): Use NVIDIA MIG to physically partition the GPU into isolated slices with dedicated hardware VRAM limits.
2. Software Memory Capping: When using NVIDIA MPS, enforce strict caps via CUDA_MPS_PINNED_STATIC_MEM_LIMIT=0=10G.

---

3. Security Checklist for Multi-Tenant GPU Clusters

Hardening Requirement	Status	Implementation
No --privileged Flags	✅ Enforced	Restrict CLI flags in CI/CD pipelines
Driver Capabilities	✅ Enforced	Set NVIDIA_DRIVER_CAPABILITIES=compute,utility
cgroups v2 Enforcement	✅ Enforced	Migrate host to Docker Engine 25+ and CDI
VRAM Zeroing	✅ Enforced	Set NVreg_InitializeSystemMemoryAllocations=1
Hardware Isolation	✅ Enforced	Use NVIDIA MIG for strict multi-tenant boundaries

---

Conclusion

Securing GPU-enabled containers requires moving away from legacy unrestricted options:

1. Never run GPU containers with --privileged.
2. Restrict container driver capabilities via NVIDIA_DRIVER_CAPABILITIES.
3. Leverage NVIDIA MIG for multi-tenant isolation at the silicon level.