How to Fix Ollama 'No GPU Detected' in Docker Containers

Overview

When running Ollama in Docker, users frequently discover that token generation is excruciatingly slow (1-3 tokens per second). Inspecting container logs reveals that Ollama has silently fallen back to CPU inference with messages like “NVIDIA GPU not detected, falling back to CPU” or “failed to initialize CUDA driver”.

This guide explains why Ollama drops back to CPU execution and provides exact Docker CLI and Docker Compose configurations to force GPU layer offloading.

Technical Causes of CPU Fallback

Ollama relies on dynamic loading of CUDA libraries (libcuda.so, libnvidia-ml.so) at runtime. If the container is started without explicit GPU device reservation or if cgroups permission isolation blocks access to /dev/nvidia* character devices, Ollama defaults to CPU-only execution.

┌────────────────────────────────────────────────────────┐
│                   OLLAMA STARTUP SEQUENCE             │
│                                                        │
│  1. Check CUDA Libraries (/usr/local/cuda/lib64)       │
│  2. Probe NVIDIA Character Devices (/dev/nvidia0)      │
│                                                        │
│         ┌───────────────────────┴───────────────────────┐
│         ▼                                               ▼
│  [ Devices Present ]                         [ Devices Missing ]
│  Load Layers onto VRAM                       FALLBACK TO CPU MODE
│  Output: 85 tok/s                            Output: 2.1 tok/s
└────────────────────────────────────────────────────────┘

Step 1: Force GPU Passthrough via Docker CLI

When running Ollama using docker run, pass --gpus=all explicitly:

docker run -d   --gpus=all   -v ollama:/root/.ollama   -p 11434:11434   --name ollama   --restart unless-stopped   ollama/ollama:latest

Step 2: Production Docker Compose Configuration

For Docker Compose v2.27+, specify the NVIDIA driver capabilities inside the deploy block:

version: "3.8"

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama_gpu
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    environment:
      - OLLAMA_KEEP_ALIVE=24h
      - OLLAMA_NUM_PARALLEL=4

volumes:
  ollama_data:

Step 3: Verification & Log Inspection

Inspect Ollama container logs during model inference:

docker logs -f ollama_gpu

Successful GPU Detection Logs:

time=2026-07-27T14:10:05Z level=INFO source=gpu.go:115 msg="NVIDIA GPU detected" count=1
time=2026-07-27T14:10:05Z level=INFO source=gpu.go:168 msg="CUDA compute capability: 8.9"
time=2026-07-27T14:10:06Z level=INFO source=server.go:210 msg="offloading 33/33 layers to GPU" VRAM=14280MB

Monitor Real-Time VRAM Usage

Run nvidia-smi on the host to confirm that memory is allocated to ollama_runner:

watch -n 1 nvidia-smi

Diagnostic Matrix

Symptom	Root Cause	Fix / Remediation
`offloading 0/33 layers to GPU`	Ollama started without `--gpus all`	Re-create container adding `--gpus=all` or Compose device block.
`cuda runtime error: out of memory`	Model parameters exceed available VRAM	Set `OLLAMA_NUM_PARALLEL=1` or load a 4-bit quantized model (e.g. `llama3:8b-instruct-q4_K_M`).
`failed to initialize nvml`	Host NVIDIA driver crash	Restart host driver: `sudo systemctl restart nvidia-persistenced`.

Summary Checklist

Passed --gpus=all or Compose device reservation.
Verified offloading 33/33 layers to GPU in logs.
Confirmed active VRAM allocation in nvidia-smi.

Root Cause Analysis & Deep Diagnostics

┌────────────────────────────────────────────────────────────┐
│                DIAGNOSTIC WORKFLOW MATRIX                  │
│                                                            │
│  1. Check Kernel & Driver State (dmesg / nvidia-smi)       │
│  2. Inspect Container Runtime Logs (docker logs -f)        │
│  3. Verify Host Network Bindings (netstat / ss -tulpn)    │
│  4. Validate Environment Variables & Secret Tokens         │
└────────────────────────────────────────────────────────────┘

Diagnostic Command Sequence

Run the following diagnostic script to isolate kernel, container, and memory bottlenecks:

#!/usr/bin/env bash
set -euo pipefail

echo "=== System Kernel & Driver Status ==="
uname -r
nvidia-smi || echo "NVIDIA Driver missing or unloaded"

echo "=== Container Health & Port Allocation ==="
docker ps --format "table {{.Names}}	{{.Status}}	{{.Ports}}"

echo "=== Memory & Swap Diagnostics ==="
free -h
swapon --show

echo "=== Systemd Service Logs ==="
journalctl -u docker -n 50 --no-pager

Comprehensive Failure Modes & Solution Matrix

Error Code / Log Entry	Underlying Root Cause	Verification Command	Resolution Procedure
`CUDA_ERROR_OUT_OF_MEMORY`	Context window exceeds allocated VRAM	`nvidia-smi`	Enable 4-bit quantization (AWQ/GPTQ) or reduce `--max-model-len`.
`docker: unknown option --gpus`	Missing NVIDIA Container Toolkit hook	`docker info \| grep Runtime`	Run `sudo nvidia-ctk runtime configure` and restart Docker.
`502 Bad Gateway`	Reverse proxy routed before backend ready	`curl -Iv http://localhost:8000`	Add container dependency healthchecks in `docker-compose.yml`.
`Permission denied (publickey)`	Incorrect SSH or token permission	`ls -la ~/.ssh/`	Fix file permissions: `chmod 600 ~/.ssh/id_rsa`.

Step-by-Step Verification Procedure

Verify Backend Availability: Ensure the backend container process is listening on the expected loopback interface using ss -tulpn | grep <port>.
Validate Payload Responses: Send a synthetic request payload using curl with verbose output -v to confirm HTTP 200 OK status.
Monitor System Telemetry: Watch memory and CPU counters in real time with htop or docker stats during peak request load.