Fix: vLLM CUDA Out of Memory (OOM) Errors: Complete Guide

Quick Fix for vLLM CUDA OOM

vLLM pre-allocates 90% of total available GPU VRAM on startup by default for its Key-Value (KV) cache. If your GPU has background processes, display servers, or your model context is too large, the startup initialization will fail.

Apply this immediate remediation to restrict memory pre-allocation and limit the context window:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3-8B-Instruct \
  --gpu-memory-utilization 0.75 \
  --max-model-len 4096 \
  --enforce-eager

---

Symptoms of vLLM CUDA OOM

1. Crash during KV Cache Profiling:

   ValueError: The model's weights and KV cache are too large for the GPU memory.
   Total GPU memory: 23.69 GiB, Model weights: 15.02 GiB, KV cache: 8.67 GiB

2. PyTorch Runtime Error:

   torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 512.00 MiB

3. Zombie VRAM Lock: nvidia-smi displays high VRAM utilization (e.g., 20GB/24GB), but no active PID is visible in standard process managers.

---

Root Causes of Memory Allocation Failures

Total GPU VRAM (e.g. 24GB RTX 4090)
┌─────────────────────────────────────────────────────────────┐
│ [1. Model Weights (~15GB)]                                  │
│ [2. PyTorch Activation Buffers (~1.5GB)]                    │
│ [3. KV Cache for Concurrent Users (~6.5GB)]                 │
│ [4. Display / System Overhead (~1.0GB)] ──▶ CRASH TRIGGER!  │
└─────────────────────────────────────────────────────────────┘

1. Default 0.90 Memory Utilization: Assumes 100% of the GPU is exclusively dedicated to vLLM with zero OS/driver overhead.
2. Unconstrained Context Length (--max-model-len): Modern models support 32k to 128k context windows. Each extra token requires proportional memory inside the KV cache blocks.
3. Ghost / Zombie CUDA Processes: Previous crashed Python or Docker instances holding onto GPU contexts without releasing CUDA handles.

---

Step-by-Step Fixes

1. Kill Zombie CUDA Processes & Free Trapped VRAM

Run this diagnostic command to find and terminate processes holding NVIDIA file descriptors:

# Check running processes attached to GPU 0
sudo fuser -v /dev/nvidia0

# Kill all dangling processes holding GPU handles
sudo fuser -k -9 /dev/nvidia*

---

2. Context Window & VRAM Sizing Reference Matrix

Before choosing your --max-model-len, check the required memory per user stream:

Model Architecture	Precision	Minimum Weights VRAM	Recommended Max Context (24GB GPU)	Recommended Max Context (80GB GPU)
Llama 3 8B	FP16 (16-bit)	~16 GB	8192	32768
Llama 3 8B	AWQ (4-bit)	~5.5 GB	16384	65536
Llama 3 70B	AWQ (4-bit)	~38 GB	Requires 2x 24GB or 1x 48GB	32768
Mistral 7B	FP16 (16-bit)	~14.5 GB	8192	32768

---

3. Tune Memory Utilization & CPU Swap Space

Configure vLLM to reserve CPU system RAM for KV cache offloading during traffic spikes:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3-8B-Instruct \
  --gpu-memory-utilization 0.80 \
  --max-model-len 8192 \
  --swap-space 8 \
  --disable-log-stats

• --gpu-memory-utilization 0.80: Restricts vLLM to use max 80% of the GPU’s memory.
• --swap-space 8: Allocates 8GB of system RAM as a fallback swap space for evicted KV blocks.

---

4. Enable AWQ / GPTQ 4-Bit Quantization

Switching from unquantized FP16 to 4-bit AWQ reduces base model weight memory by ~70% without sacrificing inference quality:

python -m vllm.entrypoints.openai.api_server \
  --model casperhansen/llama-3-8b-instruct-awq \
  --quantization awq \
  --dtype half

---

5. Multi-GPU Tensor Parallelism

If your model exceeds the capacity of a single physical card, split layers across multiple GPUs using tensor parallelism:

# Split across 2x GPUs
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3-70B-Instruct \
  --tensor-parallel-size 2

---

Prevention & Production Best Practices

1. Deploy in Headless Linux Environments: Avoid running inference on desktop operating systems (Ubuntu Desktop, Windows WSL) where Xorg/Wayland consumes 1–2GB of baseline VRAM.
2. Set Docker ipc: host: Avoid PyTorch shared-memory bus errors when sharing tensors between container workers.
3. Monitor with DCGM: Use NVIDIA DCGM Exporter and Prometheus to monitor real-time KV cache usage.

---

Related Guides & Blueprints

• Deployment Setup: How to Deploy vLLM on a VPS with Docker
• Performance Benchmarks: vLLM vs Ollama: Inference Latency & Concurrency Benchmark
• GPU Cloud Automation: RunPod Terraform Provider: Complete IaC Guide

Root Cause Analysis & Deep Diagnostics

┌────────────────────────────────────────────────────────────┐
│                DIAGNOSTIC WORKFLOW MATRIX                  │
│                                                            │
│  1. Check Kernel & Driver State (dmesg / nvidia-smi)       │
│  2. Inspect Container Runtime Logs (docker logs -f)        │
│  3. Verify Host Network Bindings (netstat / ss -tulpn)    │
│  4. Validate Environment Variables & Secret Tokens         │
└────────────────────────────────────────────────────────────┘

Diagnostic Command Sequence

Run the following diagnostic script to isolate kernel, container, and memory bottlenecks:

#!/usr/bin/env bash
set -euo pipefail

echo "=== System Kernel & Driver Status ==="
uname -r
nvidia-smi || echo "NVIDIA Driver missing or unloaded"

echo "=== Container Health & Port Allocation ==="
docker ps --format "table {{.Names}}	{{.Status}}	{{.Ports}}"

echo "=== Memory & Swap Diagnostics ==="
free -h
swapon --show

echo "=== Systemd Service Logs ==="
journalctl -u docker -n 50 --no-pager

Comprehensive Failure Modes & Solution Matrix

Error Code / Log Entry	Underlying Root Cause	Verification Command	Resolution Procedure
CUDA_ERROR_OUT_OF_MEMORY	Context window exceeds allocated VRAM	nvidia-smi	Enable 4-bit quantization (AWQ/GPTQ) or reduce --max-model-len.
docker: unknown option --gpus	Missing NVIDIA Container Toolkit hook	docker info | grep Runtime	Run sudo nvidia-ctk runtime configure and restart Docker.
502 Bad Gateway	Reverse proxy routed before backend ready	curl -Iv http://localhost:8000	Add container dependency healthchecks in docker-compose.yml.
Permission denied (publickey)	Incorrect SSH or token permission	ls -la ~/.ssh/	Fix file permissions: chmod 600 ~/.ssh/id_rsa.

---

Step-by-Step Verification Procedure

1. Verify Backend Availability: Ensure the backend container process is listening on the expected loopback interface using ss -tulpn | grep <port>.
2. Validate Payload Responses: Send a synthetic request payload using curl with verbose output -v to confirm HTTP 200 OK status.
3. Monitor System Telemetry: Watch memory and CPU counters in real time with htop or docker stats during peak request load.