Fix: Docker NVIDIA Runtime Not Found

In AI infrastructure, the most common deployment hurdle is the “GPU Passthrough.” Because Docker containers are isolated from the host hardware, they require a specific runtime shim to access the NVIDIA drivers.

Quick Fix

Run the following commands on your host to register the NVIDIA runtime with Docker:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Symptoms

When attempting to start a container with GPU access (using --gpus all or Compose reservations), you receive:

docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].

Root Causes

1. NVIDIA Container Toolkit Not Installed: The libraries that bridge Docker and the driver are missing.
2. Runtime Not Registered: The toolkit is installed, but /etc/docker/daemon.json has not been updated to include the nvidia runtime.
3. Daemon Not Restarted: Configuration changes were made, but the Docker service was not restarted.

Step-by-Step Fix

1. Verify Host Drivers

Ensure the NVIDIA drivers are working on the host:

nvidia-smi

If this fails, resolve host drivers first.

2. Install NVIDIA Container Toolkit

Add the repository and install the package:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update && sudo apt install -y nvidia-container-toolkit

3. Register the Runtime

Configure Docker to recognize the toolkit:

sudo nvidia-ctk runtime configure --runtime=docker

4. Restart Docker

Apply the changes:

sudo systemctl restart docker

Verification

Run a test container to confirm GPU access:

docker run --rm --gpus all nvidia/cuda:12.0-base-ubuntu22.04 nvidia-smi

If you see the GPU status table, the fix is successful.

Prevention

• Automation: Include the nvidia-ctk configuration in your Ansible or Terraform provisioning scripts.
• Kernel Updates: Use nvidia-dkms to ensure drivers persist through kernel updates.

Related Guides

• Blueprint: Private AI Stack Blueprint.
• Tutorial: Deploying Ollama with Docker Compose.

FAQ

Q: Does this work with Docker Compose? A: Yes. Once the runtime is registered globally, Compose can use it via the deploy.resources.reservations block.

Q: Can I use this for AMD GPUs? A: No. AMD uses the ROCm runtime, which requires different setup steps.