Secure Sovereign AI Infrastructure Blueprint: OPNsense + Docker

Executive Summary

Deploying sovereign AI infrastructure requires strict isolation between untrusted web access, internal agent communication, and GPU compute instances. Exposing model endpoints like Ollama, vLLM, or Open-WebUI directly to standard LANs introduces severe risk: prompt injection attacks, vector store exfiltration, and unmonitored host access.

This blueprint delivers an enterprise-grade architecture pattern for hosting private LLMs on bare-metal servers using OPNsense firewall micro-segmentation, isolated Docker bridge networks, and NVIDIA container runtime security controls.

High-Level Topology Blueprint

                      [ INTERNET / WAN ]
                              │
                              ▼
┌────────────────────────────────────────────────────────────┐
│                    OPNsense Gateway                        │
│  - WAN Interface: Public IP / Cloudflare Tunnel Ingress   │
│  - Suricata IDS/IPS Enabled                               │
│  - Unbound DNS with DNSBL Filtering                        │
└──────┬──────────────────────┬──────────────────────┬───────┘
       │                      │                      │
       │ VLAN 10 (Management) │ VLAN 20 (AI Apps)    │ VLAN 30 (GPU Node)
       ▼                      ▼                      ▼
┌──────────────┐      ┌──────────────┐      ┌─────────────────────────┐
│ Admin Workst │      │ Open-WebUI / │      │ Private GPU Server      │
│ 10.10.0.0/24 │      │ Agent Nodes  │      │ - vLLM / Ollama Engine  │
└──────────────┘      │ 10.20.0.0/24 │      │ - 10.30.0.0/24          │
                      └──────────────┘      └─────────────────────────┘

Core Security & Isolation Principles

Micro-Segmentation: Separate user-facing web UIs (Open-WebUI) from inference engines (vLLM/Ollama) across distinct OPNsense VLAN interfaces.
Strict Ingress Filtering: Allow TCP traffic to GPU nodes only from explicit frontend container IPs on designated ports (11434, 8000).
No Direct Internet Access for Inference Containers: Prevent model execution environments from initiating arbitrary outbound TCP/UDP connections.
Non-Root Container Execution: Enforce non-root execution and drop ALL Linux capabilities (cap_drop: - ALL) except required GPU device access.

Step-by-Step Implementation Guide

1. OPNsense Interface & VLAN Configuration

Configure three distinct virtual local area networks on your OPNsense firewall trunk port (parent: igc0):

# OPNsense Interface Mapping
VLAN 10 -> Mgmt_LAN (10.10.0.0/24)
VLAN 20 -> AI_Apps_Zone (10.20.0.0/24)
VLAN 30 -> AI_GPU_Zone (10.30.0.0/24)

Firewall Ruleset Matrix

Create the following rules under Firewall -> Rules -> AI_Apps_Zone:

# Rule 1: Allow Open-WebUI to reach vLLM engine
Action: PASS
Interface: AI_Apps_Zone (VLAN 20)
Protocol: TCP
Source: 10.20.0.10 (Open-WebUI IP)
Destination: 10.30.0.50 (GPU Node IP)
Port: 8000 (vLLM OpenAI API)

# Rule 2: Block AI Apps Zone from Management LAN
Action: BLOCK
Interface: AI_Apps_Zone (VLAN 20)
Protocol: ANY
Source: 10.20.0.0/24
Destination: 10.10.0.0/24

2. GPU Host Security & Docker Network Setup

On the target bare-metal server (10.30.0.50), configure a hardened Docker Compose deployment with strict network bindings and resource limits.

Create /etc/docker/daemon.json to enforce container security defaults:

{
  "icc": false,
  "no-new-privileges": true,
  "default-ulimits": {
    "nofile": {
      "Name": "nofile",
      "Hard": 64000,
      "Soft": 64000
    }
  },
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

Restart Docker to apply rules:

sudo systemctl restart docker

3. Production Hardened Docker Compose Stack

Deploy the core sovereign AI stack using the following hardening manifest:

version: "3.8"

services:
  vllm_engine:
    image: vllm/vllm-openai:v0.4.2
    container_name: sovereign_vllm
    restart: unless-stopped
    user: "1000:1000"
    read_only: false
    security_opt:
      - no-new-privileges:true
    cap_drop:
      - ALL
    environment:
      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}
      - TRANSFORMERS_CACHE=/root/.cache/huggingface
    volumes:
      - /opt/models/huggingface:/root/.cache/huggingface:rw
    ports:
      - "10.30.0.50:8000:8000"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    networks:
      gpu_backend:
        ipv4_address: 172.28.0.10

  open_webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: sovereign_webui
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    environment:
      - OPENAI_API_BASE_URL=http://172.28.0.10:8000/v1
      - OPENAI_API_KEY=sk-sovereign-internal-key
      - WEBUI_AUTH=true
    volumes:
      - /opt/webui_data:/app/backend/data
    ports:
      - "10.20.0.10:8080:8080"
    networks:
      gpu_backend:
        ipv4_address: 172.28.0.20

networks:
  gpu_backend:
    driver: bridge
    ipam:
      config:
        - subnet: 172.28.0.0/16

Verification & Security Auditing

1. Verify Inter-Container Network Isolation

Execute a connectivity test from the open_webui container to confirm that unauthorized ports are dropped:

docker exec -it sovereign_webui curl -I http://172.28.0.10:8000/health

Expected Output: HTTP/1.1 200 OK

2. Verify External Isolation

Attempt to reach the GPU node (10.30.0.50:8000) directly from an unauthorized device on VLAN 10 (Mgmt):

curl --connect-timeout 3 http://10.30.0.50:8000/v1/models

Expected Output: curl: (28) Connection timed out (Packet dropped by OPNsense).

Comprehensive Failure Modes & Troubleshooting

Issue / Symptom	Primary Cause	Diagnostic Command	Remediation
502 Bad Gateway in WebUI	vLLM engine initializing or VRAM exhausted	`docker logs sovereign_vllm`	Wait for weight loading or add `--gpu-memory-utilization 0.90`.
Permission Denied on Volumes	Non-root `user: 1000:1000` mismatch on model directory	`ls -ld /opt/models/huggingface`	Run `chown -R 1000:1000 /opt/models`.
NVIDIA Container Runtime Failure	Missing NVIDIA Container Toolkit hook	`nvidia-smi` inside container	Re-install `nvidia-container-toolkit` and restart Docker.
OPNsense Packet Drop	Missing outbound stateful rule	OPNsense -> Log Files -> Firewall	Ensure Stateful inspection allows established/related TCP streams.

Architectural Hardening Checklist

OPNsense VLAN isolation established (10.10.0.0, 10.20.0.0, 10.30.0.0).
Docker inter-container communication (icc) disabled globally.
Containers explicitly assigned static internal IPs on non-routable subnets.
Model cache directories stored on dedicated ZFS/NVMe arrays with strict permissions.
Firewall logging configured for all denied packets originating from AI zones.

Root Cause Analysis & Deep Diagnostics

┌────────────────────────────────────────────────────────────┐
│                DIAGNOSTIC WORKFLOW MATRIX                  │
│                                                            │
│  1. Check Kernel & Driver State (dmesg / nvidia-smi)       │
│  2. Inspect Container Runtime Logs (docker logs -f)        │
│  3. Verify Host Network Bindings (netstat / ss -tulpn)    │
│  4. Validate Environment Variables & Secret Tokens         │
└────────────────────────────────────────────────────────────┘

Diagnostic Command Sequence

Run the following diagnostic script to isolate kernel, container, and memory bottlenecks:

#!/usr/bin/env bash
set -euo pipefail

echo "=== System Kernel & Driver Status ==="
uname -r
nvidia-smi || echo "NVIDIA Driver missing or unloaded"

echo "=== Container Health & Port Allocation ==="
docker ps --format "table {{.Names}}	{{.Status}}	{{.Ports}}"

echo "=== Memory & Swap Diagnostics ==="
free -h
swapon --show

echo "=== Systemd Service Logs ==="
journalctl -u docker -n 50 --no-pager

Comprehensive Failure Modes & Solution Matrix

Error Code / Log Entry	Underlying Root Cause	Verification Command	Resolution Procedure
`CUDA_ERROR_OUT_OF_MEMORY`	Context window exceeds allocated VRAM	`nvidia-smi`	Enable 4-bit quantization (AWQ/GPTQ) or reduce `--max-model-len`.
`docker: unknown option --gpus`	Missing NVIDIA Container Toolkit hook	`docker info \| grep Runtime`	Run `sudo nvidia-ctk runtime configure` and restart Docker.
`502 Bad Gateway`	Reverse proxy routed before backend ready	`curl -Iv http://localhost:8000`	Add container dependency healthchecks in `docker-compose.yml`.
`Permission denied (publickey)`	Incorrect SSH or token permission	`ls -la ~/.ssh/`	Fix file permissions: `chmod 600 ~/.ssh/id_rsa`.

Step-by-Step Verification Procedure

Verify Backend Availability: Ensure the backend container process is listening on the expected loopback interface using ss -tulpn | grep <port>.
Validate Payload Responses: Send a synthetic request payload using curl with verbose output -v to confirm HTTP 200 OK status.
Monitor System Telemetry: Watch memory and CPU counters in real time with htop or docker stats during peak request load.