Fix: vLLM CUDA Out of Memory (OOM)

vLLM is an aggressive consumer of VRAM because it pre-allocates a large KV (Key-Value) cache to enable high-concurrency inference. Without tuning, this can lead to crashes on startup.

Quick Fix

Lower the gpu_memory_utilization parameter in your startup command (default is 0.90):

python -m vllm.entrypoints.openai.api_server \
  --model llama3-8b \
  --gpu-memory-utilization 0.70

Symptoms

• Crash on Startup: The server fails to initialize with RuntimeError: CUDA out of memory.
• KV Cache Error: Logs show ValueError: The model's weights and KV cache are too large for the GPU memory.
• ZOMBIE Process: The GPU remains fully occupied even after the process crashes.

Root Causes

1. Aggressive Pre-allocation: vLLM tries to take 90% of your VRAM for the KV cache by default.
2. Model Weights too Large: You are attempting to load a model that, even without a cache, exceeds your VRAM.
3. Context Window (Max Tokens): High context window settings (e.g., 32k or 128k) exponentially increase the KV cache size.
4. ZOMBIE VRAM: Another process (like an old Ollama instance) is already holding 2GB+ of VRAM.

Step-by-Step Fix

1. Identify Background Usage

Run nvidia-smi and look for processes. If you see ollama or another vllm process, kill it:

sudo fuser -v /dev/nvidia*
# Kill the relevant PID

2. Tune Memory Utilization

If your GPU is also driving a display or running minor background tasks, the default 90% allocation will fail. Reduce it to 70% or 80%:

--gpu-memory-utilization 0.80

3. Adjust Max Model Len

If the model’s native context window is too large for your VRAM, force it smaller:

--max-model-len 4096

4. Use Quantization

If using high-end models on consumer hardware, ensure you are using a quantized version (AWQ or SqueezeLLM):

--quantization awq

Verification

Watch the VRAM allocation during startup:

watch -n 1 nvidia-smi

The VRAM usage should jump to your specified percentage (e.g., 80% of 24GB = ~19.2GB) and remain steady without crashing.

Prevention

• Monitoring: Use prometheus and grafana to track VRAM usage and request spikes.
• Dedicated Hardware: Run production inference on “headless” servers where no GUI or X11 process consumes VRAM.

Related Guides

• Infrastructure: Sovereign AI Infrastructure Guide.
• Hardware: Best VPS Providers for AI Workloads.

FAQ

Q: Does reducing memory utilization hurt performance? A: It reduces the number of concurrent requests the server can handle because there is less room for the KV cache. It does not affect the speed of a single request.