How to Deploy vLLM on a VPS: Production Docker & CUDA Setup

Quick Answer

To deploy vLLM as an OpenAI-compatible API server in production, use the official vllm/vllm-openai:latest Docker image with GPU passthrough via docker-compose.yml:

services:
  vllm:
    image: vllm/vllm-openai:latest
    container_name: vllm-server
    restart: unless-stopped
    runtime: nvidia
    environment:
      - HUGGING_FACE_HUB_TOKEN=hf_xxxxxxxxxxxx
    volumes:
      - /root/.cache/huggingface:/root/.cache/huggingface
    ports:
      - "8000:8000"
    ipc: host
    command: >
      --model meta-llama/Meta-Llama-3-8B-Instruct
      --gpu-memory-utilization 0.90
      --max-model-len 8192
      --api-key secret_api_key_12345
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

---

Prerequisites & Host System Setup

Ensure your GPU VPS (Ubuntu 22.04 / 24.04 LTS on RunPod, Vultr, or Hetzner) has NVIDIA drivers and the container toolkit installed.

# 1. Verify NVIDIA Driver
nvidia-smi

# 2. Install NVIDIA Container Toolkit
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

---

Production Architecture

Internet / Client Apps
         │ (HTTPS Port 443 + API Key Auth)
         ▼
┌──────────────────────────────────────┐
│  Caddy / Nginx Reverse Proxy (SSL)   │
└──────────────────┬───────────────────┘
                   │ (Internal Bridge Port 8000)
                   ▼
┌──────────────────────────────────────┐
│  vLLM Engine (PagedAttention)        │
│  - Continuous Batching               │
│  - PyTorch Shared Memory (ipc: host) │
│  - GPU VRAM Allocation (90%)         │
└──────────────────┬───────────────────┘
                   │ (PCIe / NVLink)
                   ▼
┌──────────────────────────────────────┐
│  NVIDIA RTX 4090 / A100 GPU          │
└──────────────────────────────────────┘

---

Production Docker Compose with SSL Reverse Proxy

Create /opt/vllm-stack/docker-compose.yml:

version: '3.8'

services:
  caddy:
    image: caddy:2-alpine
    container_name: caddy-proxy
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy_data:/data
      - caddy_config:/config
    depends_on:
      - vllm

  vllm:
    image: vllm/vllm-openai:latest
    container_name: vllm-engine
    restart: unless-stopped
    environment:
      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}
      - VLLM_LOGGING_LEVEL=INFO
    volumes:
      - /opt/models/huggingface:/root/.cache/huggingface
    ipc: host # Prevents PyTorch Shared Memory Bus Errors
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    command: >
      --model meta-llama/Meta-Llama-3-8B-Instruct
      --dtype auto
      --gpu-memory-utilization 0.88
      --max-model-len 8192
      --api-key ${VLLM_API_KEY}
      --enforce-eager
    expose:
      - "8000"

volumes:
  caddy_data:
  caddy_config:

Caddyfile Configuration (/opt/vllm-stack/Caddyfile)

ai.yourdomain.com {
    reverse_proxy vllm:8000 {
        header_up Host {host}
        header_up X-Real-IP {remote_host}
    }
}

---

Environment File (.env)

HF_TOKEN=hf_YourHuggingFaceReadTokenHere
VLLM_API_KEY=sk_opsnexus_prod_secret_token_9981

---

Launching & Verifying the Service

# 1. Start the containers
cd /opt/vllm-stack
docker compose up -d

# 2. Stream logs to monitor model weight download and KV cache allocation
docker compose logs -f vllm

# 3. Test OpenAI API endpoint compatibility
curl https://ai.yourdomain.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk_opsnexus_prod_secret_token_9981" \
  -d '{
    "model": "meta-llama/Meta-Llama-3-8B-Instruct",
    "messages": [
      {"role": "system", "content": "You are a production infrastructure engineer."},
      {"role": "user", "content": "Give me a 1-sentence description of vLLM PagedAttention."}
    ],
    "temperature": 0.2
  }'

---

Multi-GPU & Quantization Optimizations

1. Multi-GPU Tensor Parallelism (e.g., 2x RTX 4090 or 2x A100)

When serving large models (like 70B parameters), split weights across GPUs:

--tensor-parallel-size 2

2. 4-bit AWQ / GPTQ Quantization

To fit a 70B model inside a single 24GB or 48GB GPU:

--model casperhansen/llama-3-70b-instruct-awq \
--quantization awq \
--dtype half

---

Common Pitfalls & How to Avoid Them

• Shared Memory Crashes (Bus error): Always include ipc: host in your Docker configuration. PyTorch utilizes /dev/shm for inter-process tensor transfers.
• CUDA Out of Memory during KV Cache init: If other processes consume VRAM, reduce --gpu-memory-utilization from 0.90 to 0.80. See our comprehensive guide: Fix: vLLM CUDA Out of Memory.

---

Related Guides & Blueprints

• Benchmarking: vLLM vs Ollama: Inference Latency & Concurrency Benchmark
• Troubleshooting: Fix: vLLM CUDA Out of Memory (OOM)
• Automation: RunPod Terraform Provider: Complete IaC Guide
• Hardware Guide: Best VPS Providers for Self-Hosted AI Workloads