Fix: Ollama GPU Not Detected in Docker

You have a functioning NVIDIA GPU and the NVIDIA Container Toolkit is installed, but Ollama logs show “CPU only” mode or model inference is extremely slow.

Quick Fix

Ensure your docker-compose.yml includes the deploy block and the NVIDIA_VISIBLE_DEVICES environment variable:

services:
  ollama:
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    environment:
      - NVIDIA_VISIBLE_DEVICES=all

Symptoms

• Slow Inference: Generation speed is 1-2 tokens per second (indicative of CPU inference).
• Log Messages: docker logs ollama contains llama.cpp: compute buffer: 0.00 MiB or failed to initialize library libnvidia-ml.so.
• NVIDIA-SMI: The GPU shows 0% utilization and minimal VRAM usage while Ollama is running.

Root Causes

1. Missing Reservations: Docker Compose requires an explicit deploy block to allocate the GPU.
2. Missing Environment Variables: Some Ollama images require NVIDIA_VISIBLE_DEVICES=all to see the hardware.
3. VRAM OOM (Out of Memory): If the model is too large for the VRAM, Ollama may silently fallback to CPU.
4. Driver/Toolkit Version Mismatch: The Ollama container’s CUDA version is higher than the host driver’s supported version.

Step-by-Step Fix

1. Check Compose Reservation

Verify your docker-compose.yml follows the modern specification:

    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

2. Set Environment Variables

Add these to your service definition to ensure the container can see the hardware libraries:

    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=compute,utility

3. Verify CUDA Compatibility

Check your host driver version: nvidia-smi. If you are running an older driver (e.g., 470), you may need to upgrade to 535+ to support modern LLM runtimes.

4. Check for Silent OOM

If you are trying to load a 70B model on a 24GB GPU, Ollama may fail to allocate the GPU buffers and fallback to CPU. Try loading a smaller model (e.g., llama3:8b) to see if the GPU is detected.

Verification

Run a model and watch the logs in a separate terminal:

docker logs -f ollama

Look for: llama.cpp: compute buffer: ... MiB (on GPU)

Prevention

• Baseline Test: Always run nvidia-smi inside the container first: docker exec -it ollama nvidia-smi
• Standardization: Use the official ollama/ollama:latest image to ensure library compatibility.

Related Guides

• Tutorial: Deploying Ollama with Docker Compose.
• Security: Securing AI Servers with OPNsense.

FAQ

Q: Can I limit Ollama to a specific GPU? A: Yes. In the deploy block, change count: all to device_ids: ['0'] (replacing ‘0’ with your specific GPU ID from nvidia-smi).