Fix: Ollama GPU Not Detected in Docker (CPU Fallback)

You have an NVIDIA GPU and installed the NVIDIA Container Toolkit, but Ollama logs state CPU only mode, inference is painfully slow (~1-2 tokens/sec), or nvidia-smi shows 0% GPU utilization.

---

Quick Fix: Compose-Spec GPU Field

In modern Docker Compose (v2.20+), use the standard compose-spec reservations block:

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=compute,utility
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

---

How to Check If Ollama Is Actually Using the GPU

To verify whether your model layers are offloaded to VRAM or running on CPU, use these three diagnostic checks:

1. Check with ollama ps

# Execute inside the container or host
docker exec -it ollama ollama ps

Healthy GPU Output:

NAME            ID              SIZE     PROCESSOR        UNTIL
llama3.2:3b     a849fcbf57b5    2.0 GB   100% GPU         4 minutes from now

(If PROCESSOR says 100% CPU or 50%/50% CPU/GPU, the model did not fit or CUDA initialization failed).

2. Stream Real-Time Ollama Logs

docker logs -f ollama

Look for these startup lines:

msg="source=gpu.go:199 msg=\"NVIDIA GPU detected\""
msg="source=gpu.go:204 msg=\"CUDA Compute Capability 8.9\""
msg="source=server.go:588 msg=\"llama runner started\""

3. Monitor with watch nvidia-smi

watch -n 0.5 nvidia-smi

During active token generation, the ollama_llama_server process will appear in the Processes table and power draw will jump.

---

Common Root Causes & Fixes

1. Legacy Docker Compose vs Compose-Spec

Older syntax like runtime: nvidia without the deploy.resources.reservations.devices block will fail to pass the GPU in newer Docker engines. Always use the driver: nvidia reservation block shown above.

2. NVIDIA Container Toolkit Default Runtime Not Registered

If Docker has not registered nvidia as a valid runtime, the container starts without GPU libraries.

# Register NVIDIA runtime in Docker daemon.json
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

3. Test Container GPU Access Directly

Verify if Docker can talk to the GPU independently of Ollama:

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

If this fails, reinstall the nvidia-container-toolkit package.

---

Related Guides & Blueprints

• Tutorial: Deploying Ollama with Docker Compose (Multi-GPU Guide)
• Production Serving: How to Deploy vLLM on a VPS with Docker
• OOM Prevention: Fix: vLLM CUDA Out of Memory (OOM)