Self-Hosted MLOps Stack with Docker Compose: MLflow, MinIO & Ollama

Commercial MLOps and LLMOps platforms (such as Weights & Biases, Databricks, and LangSmith) charge per-seat and per-trace fees while transmitting sensitive enterprise prompts, model weights, and benchmark evaluation data to external third-party servers.

For engineering teams with strict data residency, privacy, or air-gapped constraints, building a sovereign, self-hosted MLOps pipeline ensures 100% data ownership.

This technical guide walks through deploying a production-ready, open-source MLOps and LLM evaluation stack on Linux using Docker Compose, integrating:

1. MLflow Tracking Server: Centralized dashboard for logging prompt engineering experiments, hyperparameter tuning, metrics, and model versioning.
2. PostgreSQL: High-performance metadata store for MLflow experiments and runs.
3. MinIO (S3-Compatible Object Storage): Persistent storage for large model weights, dataset snapshots, and evaluation artifacts.
4. Ollama: Local inference engine for serving and evaluating open-weights models (e.g., Llama 3.2, Mistral).

---

High-Level MLOps Architecture

Data Scientists / CI/CD Pipelines (Python SDK)
                    │
                    ▼
┌─────────────────────────────────────────────────────────────┐
│  MLflow Tracking Server (Port 5000)                         │
│  - REST API & UI Dashboard                                  │
│  - Experiment Runs, Metrics & Artifact Registry             │
└──────────────┬──────────────────────────────┬───────────────┘
               │ (Metadata CRUD)              │ (S3 Multipart Uploads)
               ▼                              ▼
┌──────────────────────────────┐    ┌───────────────────────────┐
│ PostgreSQL 16 (Port 5432)    │    │ MinIO S3 Storage (Port 9000│
│ - Run IDs & Params           │    │ - Weights (.bin / .gguf)  │
│ - Timestamps & User Tags     │    │ - Evaluation Parquet Files│
└──────────────────────────────┘    └───────────────────────────┘
               │
               ▼ (Evaluates Generations Against)
┌─────────────────────────────────────────────────────────────┐
│  Ollama Inference Engine (Port 11434 / Local GPU)           │
│  - Executes Prompt Variations on Llama 3.2 / Mistral        │
│  - Returns Latency, Token/s & Perplexity Metrics to MLflow  │
└─────────────────────────────────────────────────────────────┘

---

Hardware & System Prerequisites

• Operating System: Ubuntu 22.04 / 24.04 LTS or Debian 12.
• Compute: Minimum 4 Dedicated vCPUs and 16GB System RAM (e.g., Hetzner CCX or Vultr Dedicated CPU).
• Storage: 100GB+ NVMe SSD storage for MinIO bucket volumes and local model caches.
• Docker: Docker Engine v24.0+ and Docker Compose v2.20+.

---

Step 1: Directory Setup & Environment Configuration

Create a dedicated directory structure on your server:

mkdir -p /opt/mlops-stack/{postgres_data,minio_data,ollama_data}
cd /opt/mlops-stack

Create /opt/mlops-stack/.env:

# PostgreSQL Metadata Configuration
POSTGRES_USER=mlflow_admin
POSTGRES_PASSWORD=generate_a_secure_postgres_password_9921
POSTGRES_DB=mlflow_db

# MinIO S3 Object Storage Credentials
MINIO_ROOT_USER=minio_admin_user
MINIO_ROOT_PASSWORD=generate_a_secure_minio_secret_key_8832
MINIO_BUCKET=mlflow-artifacts

# MLflow Master Credentials
MLFLOW_S3_ENDPOINT_URL=http://minio:9000
AWS_ACCESS_KEY_ID=minio_admin_user
AWS_SECRET_ACCESS_KEY=generate_a_secure_minio_secret_key_8832

---

Step 2: Custom MLflow Dockerfile (Dockerfile.mlflow)

The official minimal MLflow container lacks the PostgreSQL driver (psycopg2-binary) and Amazon S3 client (boto3). Build a production image with these dependencies included:

Create /opt/mlops-stack/Dockerfile.mlflow:

FROM python:3.11-slim

WORKDIR /app

# Install system utilities and database dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
    curl \
    build-essential \
    libpq-dev \
    && rm -rf /var/lib/apt/lists/*

# Install MLflow with PostgreSQL and S3 backends
RUN pip install --no-cache-dir \
    mlflow==2.15.1 \
    psycopg2-binary==2.9.9 \
    boto3==1.34.150 \
    cryptography==42.0.8

EXPOSE 5000

---

Step 3: Complete Production docker-compose.yml

Create /opt/mlops-stack/docker-compose.yml:

version: '3.8'

services:
  # 1. PostgreSQL Database for MLflow Metadata
  postgres:
    image: postgres:16-alpine
    container_name: mlops-postgres
    restart: unless-stopped
    environment:
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: ${POSTGRES_DB}
    volumes:
      - ./postgres_data:/var/lib/postgresql/data
    networks:
      - mlops_mesh
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER} -d ${POSTGRES_DB}"]
      interval: 5s
      timeout: 5s
      retries: 5

  # 2. MinIO S3-Compatible Storage for Artifacts and Model Weights
  minio:
    image: minio/minio:RELEASE.2024-08-03T04-33-23Z
    container_name: mlops-minio
    restart: unless-stopped
    command: server /data --console-address ":9001"
    environment:
      MINIO_ROOT_USER: ${MINIO_ROOT_USER}
      MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}
    ports:
      - "9000:9000" # S3 API endpoint
      - "9001:9001" # Web Console
    volumes:
      - ./minio_data:/data
    networks:
      - mlops_mesh
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 10s
      timeout: 5s
      retries: 3

  # 3. MinIO Bucket Initialization Helper
  create-bucket:
    image: minio/mc:latest
    container_name: mlops-minio-init
    depends_on:
      minio:
        condition: service_healthy
    networks:
      - mlops_mesh
    entrypoint: >
      /bin/sh -c "
      mc alias set myminio http://minio:9000 ${MINIO_ROOT_USER} ${MINIO_ROOT_PASSWORD};
      mc mb myminio/${MINIO_BUCKET} --ignore-existing;
      exit 0;
      "

  # 4. MLflow Tracking Server
  mlflow:
    build:
      context: .
      dockerfile: Dockerfile.mlflow
    container_name: mlops-mlflow
    restart: unless-stopped
    ports:
      - "5000:5000"
    environment:
      - MLFLOW_S3_ENDPOINT_URL=${MLFLOW_S3_ENDPOINT_URL}
      - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
      - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
    command: >
      mlflow server
      --backend-store-uri postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB}
      --default-artifact-root s3://${MINIO_BUCKET}/
      --host 0.0.0.0
      --port 5000
    depends_on:
      postgres:
        condition: service_healthy
      minio:
        condition: service_healthy
    networks:
      - mlops_mesh

  # 5. Ollama LLM Inference & Evaluation Engine
  ollama:
    image: ollama/ollama:latest
    container_name: mlops-ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    environment:
      - OLLAMA_HOST=0.0.0.0
      - OLLAMA_KEEP_ALIVE=24h
    volumes:
      - ./ollama_data:/root/.ollama
    networks:
      - mlops_mesh
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

networks:
  mlops_mesh:
    driver: bridge

---

Step 4: Launching and Verifying the Stack

Build the MLflow container and start all services in detached mode:

cd /opt/mlops-stack
docker compose up -d --build

Verify that all five services are active and healthy:

docker compose ps

Pull a local model into Ollama to use for benchmark experiments:

docker exec -it mlops-ollama ollama pull llama3.2:3b

---

Step 5: Logging LLM Experiments to MLflow (Python Example)

Create an automated experiment script on your local machine or data science workstation to test different system prompts against Ollama and log latency, token metrics, and outputs directly into your self-hosted MLflow instance.

Save this script as evaluate_llm_prompts.py:

import time
import requests
import mlflow
import os

# Configure MLflow to point to your self-hosted server
MLFLOW_SERVER_URL = "http://YOUR_SERVER_IP:5000"
OLLAMA_API_URL = "http://YOUR_SERVER_IP:11434/api/generate"

# Configure S3/MinIO environment for artifact uploads
os.environ["MLFLOW_S3_ENDPOINT_URL"] = "http://YOUR_SERVER_IP:9000"
os.environ["AWS_ACCESS_KEY_ID"] = "minio_admin_user"
os.environ["AWS_SECRET_ACCESS_KEY"] = "generate_a_secure_minio_secret_key_8832"

mlflow.set_tracking_uri(MLFLOW_SERVER_URL)
mlflow.set_experiment("Customer-Support-Prompt-Evaluation")

prompts_to_test = [
    {"version": "v1_concise", "system": "You are a concise enterprise support agent. Answer in under 20 words."},
    {"version": "v2_detailed", "system": "You are a thorough support specialist. Provide step-by-step numbered instructions."}
]

test_query = "How do I configure MTU bridge settings in Docker?"

for prompt_config in prompts_to_test:
    with mlflow.start_run(run_name=f"prompt_{prompt_config['version']}"):
        # Log parameters
        mlflow.log_param("model", "llama3.2:3b")
        mlflow.log_param("prompt_version", prompt_config["version"])
        mlflow.log_param("system_prompt", prompt_config["system"])

        start_time = time.time()
        
        # Dispatch request to local Ollama instance
        response = requests.post(
            OLLAMA_API_URL,
            json={
                "model": "llama3.2:3b",
                "prompt": f"{prompt_config['system']}\nUser: {test_query}",
                "stream": False
            }
        ).json()
        
        total_latency = time.time() - start_time
        
        # Calculate tokens per second
        eval_count = response.get("eval_count", 0)
        eval_duration_sec = response.get("eval_duration", 1) / 1e9
        tokens_per_sec = eval_count / eval_duration_sec if eval_duration_sec > 0 else 0

        # Log performance and evaluation metrics
        mlflow.log_metric("total_latency_seconds", round(total_latency, 3))
        mlflow.log_metric("generated_tokens", eval_count)
        mlflow.log_metric("tokens_per_second", round(tokens_per_sec, 2))

        # Save model output as an artifact in MinIO
        with open("sample_output.txt", "w") as f:
            f.write(response.get("response", ""))
        mlflow.log_artifact("sample_output.txt")

        print(f"Logged run: {prompt_config['version']} - {tokens_per_sec:.1f} tok/s in {total_latency:.2f}s")

Execute the evaluation:

pip install mlflow requests boto3
python evaluate_llm_prompts.py

Open http://YOUR_SERVER_IP:5000 to visualize run metrics, compare token throughput, and inspect generated text artifacts stored securely in MinIO.

---

Production Security & Access Control

1. Firewall Ingress: Restrict PostgreSQL port 5432 and MinIO API port 9000 from public internet exposure. Only expose the MLflow web UI (5000) and MinIO Console (9001) behind an authentication gateway.
2. Reverse Proxy & SSL: Deploy Caddy or Nginx with Let’s Encrypt SSL certificates in front of MLflow to secure credentials and token streams in transit.
3. Automate MinIO Backups: Set up periodic S3 bucket replication or BorgBackup snapshots of /opt/mlops-stack/minio_data to off-site storage.

---

Frequently Asked Questions

Q: Why use MinIO instead of saving artifacts to local disk?

A: Local disk artifact stores in MLflow make horizontal scaling difficult. Using an S3-compatible backend like MinIO allows multiple remote training workers (e.g. RunPod GPU pods, Kubernetes nodes, or local laptops) to upload artifacts to the central MLflow registry simultaneously.

Q: Can MLflow evaluate large models like Llama 3 70B?

A: Yes. If your server or attached GPU node has sufficient VRAM (48GB+ for 4-bit or 160GB for FP16), point your evaluation script to the 70B model endpoint. Review our Llama 3 70B Hardware Sizing Guide for exact memory formulas.

Q: How does this stack compare to cloud MLflow on Databricks?

A: Self-hosted MLflow provides identical core experiment tracking and model registry APIs without enterprise per-DBU compute charges or external telemetry.

---

Recommended Internal Links

1. Deploying Autonomous AI Agents with Docker Compose
   • Anchor text: Deploying Autonomous AI Agents with Docker Compose
   • Why it is relevant: Demonstrates how to build multi-agent workflows (n8n + LiteLLM) that consume models tracked in MLflow.
2. Best VPS Providers for Self-Hosted AI Workloads
   • Anchor text: Best VPS Providers for Self-Hosted AI Workloads
   • Why it is relevant: Compares cloud server options for hosting dedicated MLOps clusters.
3. vLLM vs Ollama: Inference Latency & Concurrency Benchmark
   • Anchor text: vLLM vs Ollama: Inference Latency & Concurrency Benchmark
   • Why it is relevant: Provides performance benchmarks for choosing between Ollama and vLLM as the serving backend.

---

Authoritative External Sources

1. MLflow Official Tracking Server Guide (https://mlflow.org/docs/latest/tracking/server.html): Official documentation for configuring PostgreSQL and S3 backends.
2. MinIO Docker Deployment Quickstart (https://min.io/docs/minio/container/index.html): Reference guide for MinIO container configuration.
3. Ollama API Documentation (https://github.com/ollama/ollama/blob/main/docs/api.md): First-party reference for token counts, durations, and REST payloads.

---

🏷️ SEO Metadata

• SEO Title: Self-Hosted MLOps with Docker Compose: MLflow & Ollama
• Meta Description: Build a production-ready, sovereign MLOps stack using Docker Compose. Integrates MLflow for experiment tracking, MinIO for S3 artifacts, and Ollama for LLMs.
• URL Slug: self-hosted-mlops-mlflow-ollama
• Primary Keyword: self hosted mlops stack docker compose
• Secondary Keywords: mlflow ollama docker compose, self hosted mlflow with minio, open source ai devops tools, mlops experiment tracking ollama
• Search Intent: Practical Technical Guide / Implementation Tutorial