Fix: OPNsense Blocking Docker VLAN Traffic

You followed our OPNsense hardening guide, but now your management machine cannot reach the AI WebUI, or the Docker host cannot pull model weights.

Quick Fix

Ensure you have a Pass rule on your Management/LAN interface allowing traffic to the AI_VLAN address, and check that your AI server’s gateway is set to the OPNsense VLAN IP.

Symptoms

• Connection Timeout: Browser says “Connection Timed Out” when hitting the AI IP.
• Ping Fails: You can ping other devices in the network, but the AI server is invisible.
• Firewall Logs: Firewall > Log Files > Live View shows red rows for traffic from your IP to port 3000 or 11434.

Root Causes

1. Asymmetric Routing: Traffic goes to the AI server via the LAN gateway, but the AI server tries to respond via the VLAN gateway.
2. Missing Inbound Rule: You blocked all egress from the AI_VLAN but forgot to allow ingress into it from your management subnet.
3. Outbound NAT: If the AI server needs to pull a model (even temporarily), it requires an Outbound NAT rule on the WAN interface for the new VLAN subnet.

Step-by-Step Fix

1. Check the Log Files

Go to Firewall > Log Files > Live View. Filter by the AI server’s IP. Look for the “Label” of the blocking rule. Usually, it will be Default deny / state violation.

2. Create the Inbound Pass Rule

Go to Firewall > Rules > [Your_Source_Interface] (e.g., LAN). Add a rule at the top:

• Action: Pass
• Protocol: TCP
• Source: LAN net
• Destination: AI_Node_IP
• Port: 3000 (WebUI)

3. Fix DNS Resolution

If the AI server can’t reach OPNsense for DNS, the stack will fail. In Firewall > Rules > AI_VLAN, ensure the DNS rule is above the block rule:

• Action: Pass
• Protocol: UDP
• Destination: AI_VLAN address
• Port: 53

4. Temporary Update Access

If you need to pull a model from HuggingFace/GitHub:

1. Create a temporary rule at the top of AI_VLAN rules.
2. Action: Pass, Destination: Any.
3. Pull the model.
4. Disable the rule immediately.

Verification

1. From LAN: curl -I http://10.50.0.10:3000 (Should return HTTP 200).
2. From AI Server: ping 10.50.0.1 (Should resolve to the OPNsense gateway).

Prevention

• Aliases: Use an Alias for Management_Nodes and another for AI_Services to keep your firewall rules readable.
• State Table: If you change a rule and it doesn’t seem to work, clear the states for that IP in Diagnostics > States > Reset States.

Related Guides

• Security: Securing AI Servers with OPNsense.
• Blueprint: Private AI Stack Blueprint.

FAQ

Q: Why is ping blocked even when I allow TCP? A: Ping uses the ICMP protocol, not TCP. You need an explicit ICMP rule to allow pings between subnets.

Root Cause Analysis & Deep Diagnostics

┌────────────────────────────────────────────────────────────┐
│                DIAGNOSTIC WORKFLOW MATRIX                  │
│                                                            │
│  1. Check Kernel & Driver State (dmesg / nvidia-smi)       │
│  2. Inspect Container Runtime Logs (docker logs -f)        │
│  3. Verify Host Network Bindings (netstat / ss -tulpn)    │
│  4. Validate Environment Variables & Secret Tokens         │
└────────────────────────────────────────────────────────────┘

Diagnostic Command Sequence

Run the following diagnostic script to isolate kernel, container, and memory bottlenecks:

#!/usr/bin/env bash
set -euo pipefail

echo "=== System Kernel & Driver Status ==="
uname -r
nvidia-smi || echo "NVIDIA Driver missing or unloaded"

echo "=== Container Health & Port Allocation ==="
docker ps --format "table {{.Names}}	{{.Status}}	{{.Ports}}"

echo "=== Memory & Swap Diagnostics ==="
free -h
swapon --show

echo "=== Systemd Service Logs ==="
journalctl -u docker -n 50 --no-pager

Comprehensive Failure Modes & Solution Matrix

Error Code / Log Entry	Underlying Root Cause	Verification Command	Resolution Procedure
CUDA_ERROR_OUT_OF_MEMORY	Context window exceeds allocated VRAM	nvidia-smi	Enable 4-bit quantization (AWQ/GPTQ) or reduce --max-model-len.
docker: unknown option --gpus	Missing NVIDIA Container Toolkit hook	docker info | grep Runtime	Run sudo nvidia-ctk runtime configure and restart Docker.
502 Bad Gateway	Reverse proxy routed before backend ready	curl -Iv http://localhost:8000	Add container dependency healthchecks in docker-compose.yml.
Permission denied (publickey)	Incorrect SSH or token permission	ls -la ~/.ssh/	Fix file permissions: chmod 600 ~/.ssh/id_rsa.

---

Step-by-Step Verification Procedure

1. Verify Backend Availability: Ensure the backend container process is listening on the expected loopback interface using ss -tulpn | grep <port>.
2. Validate Payload Responses: Send a synthetic request payload using curl with verbose output -v to confirm HTTP 200 OK status.
3. Monitor System Telemetry: Watch memory and CPU counters in real time with htop or docker stats during peak request load.