Fix: Docker Container Outbound HTTPS Timeout (Host Works, MTU Bridge Fix)

A Docker container cannot perform outbound HTTPS requests (e.g., curl -I https://huggingface.co or apt-get update hangs indefinitely or times out), while the host machine connects to the internet without issues. Small HTTP requests may succeed, but larger packet transfers hang after the initial TCP handshake.

---

Quick Fix: MTU Size Mismatch on docker0 Bridge

The most common root cause in cloud VPS environments (Hetzner, AWS, OpenStack, Wireguard/Tailscale VPN tunnels) is an MTU (Maximum Transmission Unit) mismatch. The host physical interface uses an MTU of 1420 or 1450, while Docker defaults to 1500, causing packet fragmentation drops during TLS key exchange.

Update Docker daemon MTU in /etc/docker/daemon.json:

{
  "mtu": 1420
}

Then restart Docker:

sudo systemctl restart docker

---

Symptoms

1. Hangs on TLS Client Hello: Running curl -v https://example.com inside the container connects (TCP SYN/ACK succeeds), but hangs immediately during * TLSv1.3 (OUT), TLS handshake, Client hello (1):.
2. Host Connection Works Perfectly: curl https://example.com executed on the host server succeeds in milliseconds.
3. Small Payloads Pass, Large Payloads Hang: ping 8.8.8.8 and curl http://example.com succeed, but downloading a 10MB file or installing packages via pip or apt stalls at 0%.

---

Root Causes

Host Interface (MTU 1420 - e.g. Wireguard / Cloud VPC)
┌─────────────────────────────────────────────────────────────┐
│  Host Network MTU: 1420 bytes                               │
└──────────────────────────────▲──────────────────────────────┘
                               │ (Packets > 1420 dropped without ICMP)
┌──────────────────────────────▼──────────────────────────────┐
│  Docker Bridge (docker0) Default MTU: 1500 bytes            │
│  * Large TLS Certificate Packet (1460 bytes) is DROPPED!    │
└─────────────────────────────────────────────────────────────┘

1. MTU Mismatch (Path MTU Blackhole): The host’s physical network or VPN tunnel uses an MTU smaller than standard Ethernet (1500). When the container sends a 1500-byte packet with the DF (Don’t Fragment) bit set, intermediate routers drop it without returning ICMP fragmentation notifications.
2. DNS Server Routing Failure: systemd-resolved on the host binds to 127.0.0.53. If Docker is not passing proper upstream DNS servers, containerized DNS queries timeout.
3. iptables FORWARD Policy Drop: The host firewall or ufw default forward policy is set to DROP, blocking container NAT packets.

---

Step-by-Step Diagnostic & Permanent Remediation

1. Identify Host Network MTU

Check the active MTU of your host server’s primary outbound interface:

ip link show | grep -E 'mtu|state UP'

Example Output:

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1420 qdisc fq_codel state UP
3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state DOWN

Notice eth0 is 1420 while docker0 is 1500.

---

2. Configure Global Docker Daemon MTU

Edit /etc/docker/daemon.json (create it if it does not exist):

{
  "mtu": 1420,
  "dns": ["1.1.1.1", "8.8.8.8"]
}

Restart the Docker daemon to apply changes to the default docker0 bridge:

sudo systemctl restart docker

---

3. Configure Custom Docker Compose Networks

If your containers run inside a custom bridge network defined in docker-compose.yml, specify the MTU explicitly in the network driver options:

version: '3.8'

services:
  ai-app:
    image: python:3.11-slim
    command: curl -Iv https://huggingface.co
    networks:
      - custom_mesh

networks:
  custom_mesh:
    driver: bridge
    driver_opts:
      com.docker.network.driver.mtu: "1420"

---

4. Fix Host iptables / UFW Forwarding Rules

If MTU matches but outbound packets are still blocked:

# Allow packet forwarding in sysctl
sudo sysctl -w net.ipv4.ip_forward=1
echo "net.ipv4.ip_forward=1" | sudo tee -a /etc/sysctl.d/99-docker-forward.conf

# If using UFW, set default forward policy to ACCEPT in /etc/default/ufw:
# DEFAULT_FORWARD_POLICY="ACCEPT"
sudo ufw reload

---

Verification Test

Test MTU packet transmission from inside a temporary container:

# 1. Test ping without fragmentation
docker run --rm alpine ping -c 3 -s 1390 -M do 1.1.1.1

# 2. Test full HTTPS TLS handshake and certificate download
docker run --rm curlimages/curl -Iv https://huggingface.co

If both commands return immediately with HTTP/2 200 or HTTP/1.1 200 OK, the MTU bottleneck is resolved.

---

Related Guides & Blueprints

• OPNsense Firewall: Fix: OPNsense Blocking Docker VLAN Traffic
• Private AI Stack: Private AI Stack Architecture Blueprint
• GPU Container Toolkit: Fix: Ollama GPU Not Detected in Docker

Root Cause Analysis & Deep Diagnostics

┌────────────────────────────────────────────────────────────┐
│                DIAGNOSTIC WORKFLOW MATRIX                  │
│                                                            │
│  1. Check Kernel & Driver State (dmesg / nvidia-smi)       │
│  2. Inspect Container Runtime Logs (docker logs -f)        │
│  3. Verify Host Network Bindings (netstat / ss -tulpn)    │
│  4. Validate Environment Variables & Secret Tokens         │
└────────────────────────────────────────────────────────────┘

Diagnostic Command Sequence

Run the following diagnostic script to isolate kernel, container, and memory bottlenecks:

#!/usr/bin/env bash
set -euo pipefail

echo "=== System Kernel & Driver Status ==="
uname -r
nvidia-smi || echo "NVIDIA Driver missing or unloaded"

echo "=== Container Health & Port Allocation ==="
docker ps --format "table {{.Names}}	{{.Status}}	{{.Ports}}"

echo "=== Memory & Swap Diagnostics ==="
free -h
swapon --show

echo "=== Systemd Service Logs ==="
journalctl -u docker -n 50 --no-pager

Comprehensive Failure Modes & Solution Matrix

Error Code / Log Entry	Underlying Root Cause	Verification Command	Resolution Procedure
CUDA_ERROR_OUT_OF_MEMORY	Context window exceeds allocated VRAM	nvidia-smi	Enable 4-bit quantization (AWQ/GPTQ) or reduce --max-model-len.
docker: unknown option --gpus	Missing NVIDIA Container Toolkit hook	docker info | grep Runtime	Run sudo nvidia-ctk runtime configure and restart Docker.
502 Bad Gateway	Reverse proxy routed before backend ready	curl -Iv http://localhost:8000	Add container dependency healthchecks in docker-compose.yml.
Permission denied (publickey)	Incorrect SSH or token permission	ls -la ~/.ssh/	Fix file permissions: chmod 600 ~/.ssh/id_rsa.

---

Step-by-Step Verification Procedure

1. Verify Backend Availability: Ensure the backend container process is listening on the expected loopback interface using ss -tulpn | grep <port>.
2. Validate Payload Responses: Send a synthetic request payload using curl with verbose output -v to confirm HTTP 200 OK status.
3. Monitor System Telemetry: Watch memory and CPU counters in real time with htop or docker stats during peak request load.