How to Deploy vLLM on VPS with Docker for High-Throughput LLM Serving

Overview

vLLM is an open-source, high-throughput LLM serving engine powered by PagedAttention—a memory management algorithm that drastically reduces VRAM fragmentation during model inference. Compared to standard Hugging Face Transformers or naive PyTorch deployments, vLLM delivers up to 14x higher request throughput.

This guide provides a production-ready step-by-step procedure for deploying vLLM on cloud GPU VPS providers (such as RunPod, Vultr GPU, or Lambda Labs) using Docker Compose, optimized CUDA parameters, and API token security.

Prerequisites & Minimum System Specifications

To achieve high-throughput token generation without out-of-memory crashes, verify your GPU VPS instance satisfies the following requirements:

Step 1: Install NVIDIA Container Toolkit

Before launching vLLM containers, install and configure the NVIDIA Container Toolkit to grant Docker access to hardware GPU acceleration.

# Update repository index and install dependencies
sudo apt-get update && sudo apt-get install -y curl gpg

# Add NVIDIA Container Toolkit repository
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list |   sed 's#deb [^ ]*#& [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg]#g' |   sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

# Install toolkit and restart Docker service
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Verify GPU availability in Docker:

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

Step 2: Production Docker Compose Deployment

Create a dedicated directory for your vLLM deployment:

mkdir -p /opt/vllm-server && cd /opt/vllm-server

Create /opt/vllm-server/docker-compose.yml:

version: "3.8"

services:
  vllm:
    image: vllm/vllm-openai:v0.4.2
    container_name: vllm_inference_engine
    restart: unless-stopped
    ipc: host
    environment:
      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}
      - VLLM_ATTENTION_BACKEND=FLASH_ATTN
    volumes:
      - /opt/vllm-server/models:/root/.cache/huggingface:rw
    ports:
      - "8000:8000"
    command: >
      --model meta-llama/Meta-Llama-3-8B-Instruct
      --port 8000
      --max-model-len 8192
      --gpu-memory-utilization 0.90
      --tensor-parallel-size 1
      --api-key ${VLLM_API_KEY}
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Create a .env file containing your secret credentials:

HF_TOKEN=hf_your_huggingface_access_token
VLLM_API_KEY=sk-opsnexus-vllm-secure-key-998

Step 3: Launch & Monitor Container Execution

Start the container in detached mode:

docker compose up -d

Monitor model weights download and engine initialization logs:

docker logs -f vllm_inference_engine

Successful Startup Indicators:

INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

Step 4: Validate OpenAI API Endpoint

Test the vLLM OpenAI-compatible endpoint using curl:

curl http://localhost:8000/v1/chat/completions   -H "Content-Type: application/json"   -H "Authorization: Bearer sk-opsnexus-vllm-secure-key-998"   -d '{
    "model": "meta-llama/Meta-Llama-3-8B-Instruct",
    "messages": [
      {"role": "system", "content": "You are a specialized AI systems engineer."},
      {"role": "user", "content": "Explain PagedAttention in two concise sentences."}
    ],
    "temperature": 0.2,
    "max_tokens": 150
  }'

Performance Tuning & Optimization Guidelines

1. Adjusting --gpu-memory-utilization

By default, vLLM pre-allocates 90% (0.90) of total VRAM for KV-cache blocks. If running alongside secondary containers (like Open-WebUI or vector DBs), reduce this value to 0.80 or 0.85 to avoid CUDA OOM crashes.

2. Multi-GPU Tensor Parallelism (--tensor-parallel-size)

For multi-GPU hosts (e.g., 2x RTX 4090s or 4x A100s), scale tensor parallelism across available GPUs:

    command: >
      --model meta-llama/Meta-Llama-3-70B-Instruct
      --tensor-parallel-size 2
      --gpu-memory-utilization 0.92

Failure Modes & Diagnostic Matrix

Summary Checklist

• Installed NVIDIA Container Toolkit and verified GPU passthrough.
• Configured ipc: host and flash-attention backend.
• Enforced API authentication via --api-key.
• Tuned memory allocation parameters for production stability.

Single-Node VPS GPU Deployment Topology

Deploying vLLM on cloud GPU VPS instances (e.g. RunPod, Lambda Labs, AWS g5.xlarge) requires balancing constrained system RAM and VRAM allocation while exposing authenticated REST endpoints.

+-------------------------------------------------------------------------+
|                    Single-Node VPS vLLM Deployment                      |
|  +-------------------------------------------------------------------+  |
|  | Caddy Edge Server (HTTPS / API Key Auth / TLS Termination)       |  |
|  +-------------------------------------------------------------------+  |
|                                   || Internal HTTP (127.0.0.1:8000)     |
|                                   \/                                    |
|  +-------------------------------------------------------------------+  |
|  | vLLM Engine Container (NVIDIA RTX 4090 / L4 / A10G GPU)           |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

Production Docker Compose Stack (docker-compose.yml)

version: "3.8"

services:
  caddy:
    image: caddy:2-alpine
    container_name: vllm-caddy-proxy
    restart: always
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - caddy_data:/data

  vllm:
    image: vllm/vllm-openai:latest
    container_name: vllm-server
    restart: always
    runtime: nvidia
    environment:
      - CUDA_VISIBLE_DEVICES=0
      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}
    ipc: host
    expose:
      - "8000"
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    command: >
      --model Qwen/Qwen2.5-Coder-14B-Instruct
      --gpu-memory-utilization 0.90
      --max-model-len 8192

volumes:
  caddy_data:

Caddyfile Configuration (Caddyfile)

vllm.yourdomain.com {
    reverse_proxy vllm:8000

    @unauthorized {
        not header Authorization "Bearer YOUR_SECRET_API_KEY_HERE"
    }
    respond @unauthorized "Unauthorized Access Denied" 401
}