How to Fix 'docker: Error response from daemon: unknown option --gpus' & NVIDIA Runtime Errors

Overview

When launching GPU-accelerated Docker containers using --gpus all or deploy.resources.reservations.devices in Docker Compose, users frequently encounter daemon errors such as:

docker: Error response from daemon: unknown option --gpus

docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]]

This guide details the underlying mechanics of the NVIDIA Container Toolkit (formerly nvidia-docker2), providing verified resolution commands for Linux distributions running Docker Engine 24.x, 25.x, and 26.x.

Root Cause Analysis

Docker Engine cannot interface directly with raw kernel GPU device drivers without a specialized runtime wrapper. The NVIDIA Container Toolkit registers nvidia-container-runtime as an OCI-compliant runtime hook inside /etc/docker/daemon.json.

If this file lacks the runtime definition—or if the Docker daemon has not been restarted after driver updates—Docker fails to recognize GPU capabilities.

┌────────────────────────────────────────────────────────┐
│                    DOCKER DAEMON                       │
│                                                        │
│  User Request: 'docker run --gpus all'                │
│                                                        │
│  ┌──────────────────────────────────────────────────┐  │
│  │ OCI Runtime Check: '/etc/docker/daemon.json'    │  │
│  └────────────────────────┬─────────────────────────┘  │
│                           │                            │
│           ┌───────────────┴───────────────┐            │
│           ▼                               ▼            │
│    [ Missing Hook ]              [ Hook Configured ]   │
│    CRASH: Unknown option          Routes to libnvidia   │
└────────────────────────────────────────────────────────┘

Step-by-Step Resolution Procedure

Step 1: Verify Host NVIDIA Kernel Driver

Before configuring Docker, verify that the host operating system recognizes the installed NVIDIA graphics hardware and kernel module:

nvidia-smi

If nvidia-smi outputs NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver, install or reload the host driver:

sudo apt-get update
sudo apt-get install -y nvidia-driver-535
sudo reboot

Step 2: Install NVIDIA Container Toolkit Packages

Ensure the repository signing keys and container toolkit components are installed:

# Setup repository key ring
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list |   sed 's#deb [^ ]*#& [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg]#g' |   sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

# Install packages
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

Step 3: Configure Docker Daemon Runtime

Automatically inject the NVIDIA OCI runtime configuration into /etc/docker/daemon.json:

sudo nvidia-ctk runtime configure --runtime=docker

Manually verify the contents of /etc/docker/daemon.json:

{
  "runtimes": {
    "nvidia": {
      "args": [],
      "path": "nvidia-container-runtime"
    }
  },
  "default-runtime": "nvidia"
}

Restart the Docker daemon to reload configuration:

sudo systemctl restart docker

Step 4: Verification & Smoke Test

Execute an isolated CUDA verification container to confirm hardware passthrough:

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

Expected Successful Output:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.154.05              Driver Version: 535.154.05    CUDA Version: 12.2     |
|-------------------------------------+------------------------+--------------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC     |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M.     |
|=====================================+========================+==========================|
|   0  NVIDIA RTX 4090                Off | 00000000:01:00.0  Off |                  N/A     |
| 30%   38C    P8              18W / 450W |      0MiB / 24564MiB |      0%      Default     |
+-------------------------------------+------------------------+--------------------------+

Troubleshooting Matrix

Error Message	Root Cause	Solution
`could not select device driver ""`	Missing NVIDIA OCI runtime hook	Run `sudo nvidia-ctk runtime configure --runtime=docker` and restart Docker.
`NVIDIA-SMI failed because driver module not loaded`	Kernel update invalidated module	Execute `sudo modprobe nvidia` or reinstall `nvidia-driver-535`.
`docker compose: devices reservation ignored`	Docker Compose spec missing GPU block	Use `deploy.resources.reservations.devices` format in Compose v2.

Summary Checklist

Verified nvidia-smi works on bare-metal host.
Installed nvidia-container-toolkit.
Configured /etc/docker/daemon.json with OCI runtime.
Restarted docker.service and validated container execution.

Root Cause Analysis & Deep Diagnostics

┌────────────────────────────────────────────────────────────┐
│                DIAGNOSTIC WORKFLOW MATRIX                  │
│                                                            │
│  1. Check Kernel & Driver State (dmesg / nvidia-smi)       │
│  2. Inspect Container Runtime Logs (docker logs -f)        │
│  3. Verify Host Network Bindings (netstat / ss -tulpn)    │
│  4. Validate Environment Variables & Secret Tokens         │
└────────────────────────────────────────────────────────────┘

Diagnostic Command Sequence

Run the following diagnostic script to isolate kernel, container, and memory bottlenecks:

#!/usr/bin/env bash
set -euo pipefail

echo "=== System Kernel & Driver Status ==="
uname -r
nvidia-smi || echo "NVIDIA Driver missing or unloaded"

echo "=== Container Health & Port Allocation ==="
docker ps --format "table {{.Names}}	{{.Status}}	{{.Ports}}"

echo "=== Memory & Swap Diagnostics ==="
free -h
swapon --show

echo "=== Systemd Service Logs ==="
journalctl -u docker -n 50 --no-pager

Comprehensive Failure Modes & Solution Matrix

Error Code / Log Entry	Underlying Root Cause	Verification Command	Resolution Procedure
`CUDA_ERROR_OUT_OF_MEMORY`	Context window exceeds allocated VRAM	`nvidia-smi`	Enable 4-bit quantization (AWQ/GPTQ) or reduce `--max-model-len`.
`docker: unknown option --gpus`	Missing NVIDIA Container Toolkit hook	`docker info \| grep Runtime`	Run `sudo nvidia-ctk runtime configure` and restart Docker.
`502 Bad Gateway`	Reverse proxy routed before backend ready	`curl -Iv http://localhost:8000`	Add container dependency healthchecks in `docker-compose.yml`.
`Permission denied (publickey)`	Incorrect SSH or token permission	`ls -la ~/.ssh/`	Fix file permissions: `chmod 600 ~/.ssh/id_rsa`.

Step-by-Step Verification Procedure

Verify Backend Availability: Ensure the backend container process is listening on the expected loopback interface using ss -tulpn | grep <port>.
Validate Payload Responses: Send a synthetic request payload using curl with verbose output -v to confirm HTTP 200 OK status.
Monitor System Telemetry: Watch memory and CPU counters in real time with htop or docker stats during peak request load.