Setting Up Zero-Trust WireGuard Site-to-Site Tunnels for Remote GPU Clusters in OPNsense

Overview

Hosting GPU infrastructure on remote cloud VPS providers (such as RunPod, Vast.ai, or Lambda Labs) while running web UIs or application servers in local homelabs introduces data privacy and latency challenges. Exposing GPU endpoints directly to the public internet via raw IP addresses or open ports is an unacceptable security risk.

This guide details how to configure a Zero-Trust WireGuard Site-to-Site VPN tunnel between an OPNsense firewall gateway and a remote bare-metal GPU server, ensuring encrypted, low-latency communication for private LLM inference.

---

Network Architecture & Tunnel Topology

┌────────────────────────────────────────────────────────┐
│                   LOCAL HOMELAB ZONE                   │
│  - OPNsense Gateway (10.10.0.1)                        │
│  - Application Nodes (10.10.0.0/24)                    │
└─────────────────────────┬──────────────────────────────┘
                          │ Encrypted WireGuard UDP:51820
                          ▼
┌────────────────────────────────────────────────────────┐
│                   REMOTE GPU VPS NODE                  │
│  - Public IP: 198.51.100.50                            │
│  - WireGuard Tunnel IP: 10.99.0.2/30                   │
│  - vLLM / Inference Container (10.99.0.2:8000)         │
└────────────────────────────────────────────────────────┘

---

Step 1: Configure OPNsense WireGuard Instance

1. Enable WireGuard Plugin

Navigate to System -> Firmware -> Plugins and install os-wireguard.

2. Generate Local OPNsense Keys

Navigate to VPN -> WireGuard -> Configuration -> Local:

• Name: WG_GPU_TUNNEL
• Listen Port: 51820
• Tunnel Address: 10.99.0.1/30

Click Generate Keys and copy the Public Key.

---

Step 2: Configure Remote GPU Node (Ubuntu Linux)

Install WireGuard on the remote GPU server:

sudo apt-get update && sudo apt-get install -y wireguard

Create /etc/wireguard/wg0.conf:

[Interface]
PrivateKey = <REMOTE_GPU_PRIVATE_KEY>
Address = 10.99.0.2/30
ListenPort = 51820

[Peer]
PublicKey = <OPNSENSE_PUBLIC_KEY>
Endpoint = <YOUR_HOMELAB_PUBLIC_IP>:51820
AllowedIPs = 10.10.0.0/24, 10.99.0.1/32
PersistentKeepalive = 25

Start and enable WireGuard:

sudo systemctl enable --now wg-quick@wg0

---

Step 3: Add OPNsense Peer & Firewall Rules

1. Add Endpoint Peer in OPNsense

Navigate to VPN -> WireGuard -> Configuration -> Endpoint:

• Name: Remote_GPU_Server
• Public Key: <REMOTE_GPU_PUBLIC_KEY>
• Endpoint Address: 198.51.100.50
• Endpoint Port: 51820
• Allowed IPs: 10.99.0.2/32

2. Configure Firewall Rules

Navigate to Firewall -> Rules -> WireGuard:

• Action: PASS
• Protocol: TCP
• Source: 10.10.0.0/24 (Local Apps)
• Destination: 10.99.0.2 (GPU Node)
• Port: 8000 (vLLM API)

---

Step 4: Verification & Performance Tuning

1. Test Handshake & Ping

On the GPU host, verify active handshake:

sudo wg show wg0
ping -c 4 10.99.0.1

2. Verify Latency & Throughput

Run iperf3 over the WireGuard interface to measure encryption overhead:

# On OPNsense shell / Local host:
iperf3 -c 10.99.0.2 -p 5201

---

Diagnostic Matrix

Error Symptom	Cause	Solution
Handshake time > 2 minutes	UDP 51820 blocked by ISP or cloud provider	Open UDP port 51820 in cloud firewall (Security Groups).
Ping works, HTTP timeout on 8000	vLLM bound to 127.0.0.1 instead of 0.0.0.0	Bind vLLM container to 0.0.0.0 or 10.99.0.2.
Packet loss under heavy LLM response	MTU size mismatch causing fragmentation	Add MTU = 1360 to /etc/wireguard/wg0.conf.

1. Host Preparation & Kernel Optimization

Configure host kernel parameters to support high-throughput network sockets and memory allocation for containerized workloads. Edit /etc/sysctl.d/99-sovereign-ai.conf:

# Sovereign AI Host Tuning Parameters
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535
fs.file-max = 2097152
vm.max_map_count = 262144

Apply sysctl changes:

sudo sysctl --system

2. Production Docker Service Specification

Deploy the container stack using hardened runtime security parameters, non-root user mappings, and explicit resource allocations:

version: "3.8"

services:
  core_workload:
    image: opsnexus/engine:v1.4
    container_name: sovereign_core_workload
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    cap_drop:
      - ALL
    cap_add:
      - SYS_ADMIN
    environment:
      - LOG_LEVEL=info
      - METRICS_ENABLED=true
    volumes:
      - /opt/data:/app/data:rw
    ports:
      - "127.0.0.1:8080:8080"
    deploy:
      resources:
        limits:
          cpus: '8.0'
          memory: 32G
        reservations:
          cpus: '4.0'
          memory: 16G

---

Failure Mode 1: High VRAM Fragmentation under Continuous Batching

During extended execution, dynamic memory allocation can create fragmented VRAM memory blocks, causing unexpected out-of-memory errors even when reported total free memory appears sufficient.

• Diagnostic Command: Run nvidia-smi --query-gpu=memory.used,memory.free --format=csv every second during peak request surges.
• Resolution: Enable unified memory allocators or set PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True".

Failure Mode 2: Container Bridge Network Throughput Degradation

Heavy payload transfers between containerized proxy sidecars and backend engines can saturate default Docker bridge network buffers.

• Diagnostic Command: Test inter-container bandwidth using iperf3 between connected container sockets.
• Resolution: Tune network socket buffer sizes or utilize host networking mode (network_mode: "host") for low-latency internal traffic.

---

Multi-Site GPU Cluster Networking Architecture

When distributed AI training or inference tasks span across hybrid locations (e.g. on-premise GPU nodes and cloud bare-metal servers), interconnecting nodes over unencrypted public internet creates severe security vulnerabilities.

By deploying OPNsense with kernel-accelerated WireGuard site-to-site tunnels, all GPU node-to-node communication (Ray control frames, vLLM gRPC requests, Prometheus DCGM metrics) is encapsulated inside a high-throughput ChaCha20-Poly1350 encrypted tunnel running at line speed.

+-------------------------------------------------------------------------+
|                  Site-to-Site WireGuard GPU Interconnect                |
|  +---------------------------+           +---------------------------+  |
|  | On-Prem GPU Cluster       |           | Cloud GPU Cluster         |  |
|  | OPNsense Firewall Gateway | <=======> | OPNsense Firewall Gateway |  |
|  | Subnet: 192.168.10.0/24   | WireGuard | Subnet: 10.200.10.0/24    |  |
|  +---------------------------+ Tunnel    +---------------------------+  |
+-------------------------------------------------------------------------+

WireGuard Server Configuration File (/etc/wireguard/wg0.conf)

[Interface]
Address = 10.100.0.1/24
PrivateKey = ON_PREM_PRIVATE_KEY_HERE
ListenPort = 51820
PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

# Cloud GPU Cluster Peer
[Peer]
PublicKey = CLOUD_GPU_PUBLIC_KEY_HERE
Endpoint = 198.51.100.50:51820
AllowedIPs = 10.200.10.0/24, 10.100.0.2/32
PersistentKeepalive = 25

Tunnel Verification Script

# Verify WireGuard handshakes and transfer rates
sudo wg show

# Test latency across private tunnel
ping -c 5 10.100.0.2