Deploying Ollama with Docker Compose: A Production Guide

Quick Answer

To deploy Ollama with Docker Compose, you need a docker-compose.yml that mounts the model volume and reserves the NVIDIA GPU. Run docker compose up -d and then use docker exec -it ollama ollama run llama3 to pull and start your first model.

Introduction

Ollama has become the “standard runtime” for self-hosted AI because of its simplicity and efficient VRAM management. While the standalone binary is excellent for macOS/Windows development, a Docker Compose deployment is the correct path for server environments where reliability and reproducibility are required.

This guide walks through a production-oriented deployment that includes Open WebUI for a ChatGPT-like interface.

Prerequisites

• Host OS: Ubuntu 22.04 LTS (Recommended)
• GPU: NVIDIA GPU with 8GB+ VRAM
• Drivers: NVIDIA Driver 535+ installed on the host
• Docker: Docker Engine 24.0+ and Docker Compose v2.20+

Step 1: Install NVIDIA Container Toolkit

Docker cannot communicate with your GPU hardware by default. You must install the toolkit to bridge the gap.

# Configure the repository
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

# Install and restart Docker
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Step 2: The Production Compose File

Create a directory (e.g., ~/ai-stack) and save the following as docker-compose.yml.

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    volumes:
      - ./ollama_data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    environment:
      - OLLAMA_KEEP_ALIVE=24h # Keep models in VRAM for performance

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    ports:
      - "3000:8080"
    volumes:
      - ./webui_data:/app/backend/data
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    depends_on:
      - ollama

Step 3: Launch and Model Pull

Start the stack:

docker compose up -d

Pull a model (e.g., Llama 3 8B):

docker exec -it ollama ollama run llama3

Verification & Testing

To confirm your deployment is production-ready, perform these checks:

1. GPU Passthrough: Run docker exec -it ollama nvidia-smi. If you see your GPU listed inside the container, the toolkit is active.
2. Web UI Connectivity: Navigate to http://<your-server-ip>:3000. You should be prompted to create an admin account.
3. Internal API: From the host, run curl http://localhost:3000/ollama/api/tags. This verifies the WebUI proxy is correctly talking to the Ollama container.

Architecture Notes: Why This Setup?

1. Volume Mapping: By mapping ./ollama_data, your downloaded models survive container restarts and updates.
2. Internal Networking: Ollama and Open WebUI communicate via Docker’s internal DNS (http://ollama:11434), keeping the API hidden from the public internet.
3. Keep-Alive: Setting OLLAMA_KEEP_ALIVE=24h prevents the model from being unloaded from VRAM, ensuring the first request of the day is as fast as the last.

Security Notes

• API Access: By default, this setup exposes the Open WebUI on port 3000. Use a firewall (UFW or OPNsense) to restrict access to your local IP range.
• Rootless Docker: For enhanced security, consider running Docker in rootless mode, though this requires additional configuration for GPU passthrough.

Common Failure Modes

• “could not select device driver with capabilities: gpu”: This means the NVIDIA Container Toolkit is missing or not configured in Docker. Fix: Re-run the nvidia-ctk runtime configure command.
• Port 11434 Bind Failure: Another instance of Ollama (perhaps the native binary) is already running. Fix: Stop the local service with sudo systemctl stop ollama.

FAQ

Q: How do I update Ollama? A: Run docker compose pull && docker compose up -d. Your models will remain safe in the mapped volume.

Q: Can I run multiple models at once? A: Yes, if you have enough VRAM. Ollama will manage the swap, but for the best experience, ensure your total model sizes don’t exceed your total VRAM.

Next Steps

For a fully hardened enterprise deployment, see our Sovereign AI Infrastructure Guide.

Related Content

• Blueprint: Private AI Stack Blueprint.
• Security: Securing AI Servers with OPNsense.
• Troubleshooting: Fix: Docker NVIDIA Runtime Not Found.

Disclosure: For teams requiring high-performance GPU instances to run this stack, consider evaluating RunPod or Vultr. Both provide technical environments suitable for Docker-based AI workloads.

Root Cause Analysis & Deep Diagnostics

┌────────────────────────────────────────────────────────────┐
│                DIAGNOSTIC WORKFLOW MATRIX                  │
│                                                            │
│  1. Check Kernel & Driver State (dmesg / nvidia-smi)       │
│  2. Inspect Container Runtime Logs (docker logs -f)        │
│  3. Verify Host Network Bindings (netstat / ss -tulpn)    │
│  4. Validate Environment Variables & Secret Tokens         │
└────────────────────────────────────────────────────────────┘

Diagnostic Command Sequence

Run the following diagnostic script to isolate kernel, container, and memory bottlenecks:

#!/usr/bin/env bash
set -euo pipefail

echo "=== System Kernel & Driver Status ==="
uname -r
nvidia-smi || echo "NVIDIA Driver missing or unloaded"

echo "=== Container Health & Port Allocation ==="
docker ps --format "table {{.Names}}	{{.Status}}	{{.Ports}}"

echo "=== Memory & Swap Diagnostics ==="
free -h
swapon --show

echo "=== Systemd Service Logs ==="
journalctl -u docker -n 50 --no-pager

Comprehensive Failure Modes & Solution Matrix

Step-by-Step Verification Procedure

1. Verify Backend Availability: Ensure the backend container process is listening on the expected loopback interface using ss -tulpn | grep <port>.
2. Validate Payload Responses: Send a synthetic request payload using curl with verbose output -v to confirm HTTP 200 OK status.
3. Monitor System Telemetry: Watch memory and CPU counters in real time with htop or docker stats during peak request load.