How to Fix CUDA Out of Memory (OOM) Errors in LLM Inference

Overview

CUDA Out of Memory (torch.cuda.OutOfMemoryError) is the most frequent runtime crash encountered when deploying LLMs on consumer GPUs or cloud VPS nodes. These errors occur when the combined memory footprint of Model Weights, KV-Cache (Context Windows), and Activation Tensors exceeds total available GPU VRAM.

This guide details the mathematical memory formula for LLM inference, step-by-step diagnostic workflows, and proven quantization strategies (AWQ, GPTQ, GGUF) to eliminate OOM crashes.

---

The LLM VRAM Memory Equation

To prevent OOM errors, calculate your model’s memory footprint before deployment using this formula:

$$\text{Total VRAM Required} = \left( \frac{\text{Parameters (Billions)} \times \text{Bytes per Param}}{1} \right) + \text{KV-Cache Footprint} + \text{Activation Overhead (20%)}$$

Precision Memory Scaling Matrix

Quantization / Precision	Bytes per Parameter	VRAM Needed for 8B Model	VRAM Needed for 70B Model
16-bit (FP16 / BF16)	2 Bytes	~16 GB	~140 GB
8-bit (INT8 / FP8)	1 Byte	~8 GB	~70 GB
4-bit (AWQ / GPTQ / GGUF Q4_K_M)	0.5 Bytes	~4.5 GB	~38 GB

---

Diagnostic Workflow for CUDA OOM

When a container crashes with torch.cuda.OutOfMemoryError: CUDA out of memory, run PyTorch and CUDA diagnostic logs to pinpoint the allocation spike:

import torch

def print_gpu_memory_stats():
    if torch.cuda.is_available():
        allocated = torch.cuda.memory_allocated() / (1024**3)
        reserved = torch.cuda.memory_reserved() / (1024**3)
        max_allocated = torch.cuda.max_memory_allocated() / (1024**3)
        print(f"Allocated Memory: {allocated:.2f} GB")
        print(f"Reserved Memory:  {reserved:.2f} GB")
        print(f"Max Peak Memory:  {max_allocated:.2f} GB")

print_gpu_memory_stats()

---

Step-by-Step Fixes to Eliminate OOM

Strategy 1: Switch to 4-bit AWQ or GPTQ Quantization

For vLLM and Hugging Face deployments, replace unquantized FP16 weights with Activation-aware Weight Quantization (AWQ). AWQ retains 99%+ FP16 accuracy while reducing memory overhead by 75%.

vLLM Execution Command:

python3 -m vllm.entrypoints.openai.api_server   --model casperhansen/llama-3-70b-instruct-awq   --quantization awq   --max-model-len 4096   --gpu-memory-utilization 0.90

---

Strategy 2: Tune KV-Cache Allocation & Max Model Length

In engines like vLLM or TGI, long prompt context windows rapidly consume VRAM during continuous batching. Reduce --max-model-len or limit KV-cache pre-allocation:

# docker-compose.yml tuning snippet
services:
  vllm:
    image: vllm/vllm-openai:latest
    command: >
      --model meta-llama/Meta-Llama-3-8B-Instruct
      --gpu-memory-utilization 0.82
      --max-model-len 4096
      --max-num-seqs 64

---

Strategy 3: Enable PyTorch Memory De-fragmentation

PyTorch frequently reserves large blocks of contiguous CUDA memory that become fragmented during tensor dynamic allocation. Enable PyTorch allocator environment variables:

export PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True,max_split_size_mb:512"

In Docker Compose:

    environment:
      - PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True,max_split_size_mb:512

---

Performance Comparison: Unquantized vs AWQ Quantized

Metric	FP16 Base (Llama 3 8B)	AWQ 4-Bit (Llama 3 8B)	Improvement
Model Weight Size	15.2 GB	4.6 GB	70% Savings
Max Batch Size (24GB VRAM)	4 Streams	32 Streams	8x Throughput
Token Generation Speed	42 tok/s	68 tok/s	+61% Speed
MMLU Accuracy Score	66.8%	66.4%	<0.5% Loss

---

Failure Modes & Quick Remediation

Symptom	Root Cause	Solution
CUDA out of memory during backward pass	Batch size too large during fine-tuning	Enable gradient_accumulation_steps and gradient_checkpointing.
OOM during prompt evaluation phase	Context length exceeds allocated KV memory	Set --max-model-len 2048 or use chunked prefill (--enable-chunked-prefill).
VRAM usage slowly increases over time	Memory leak in Custom PyTorch extensions	Enforce torch.cuda.empty_cache() at end of long generation streams.

---

Final Summary

1. Calculate VRAM Requirements: Ensure Weight Size + KV Cache + 20% Overhead <= Total VRAM.
2. Use 4-Bit AWQ: Drastically shrink model size without sacrificing intelligence.
3. Set Allocator Flags: Use PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True".

Root Cause Analysis & Deep Diagnostics

┌────────────────────────────────────────────────────────────┐
│                DIAGNOSTIC WORKFLOW MATRIX                  │
│                                                            │
│  1. Check Kernel & Driver State (dmesg / nvidia-smi)       │
│  2. Inspect Container Runtime Logs (docker logs -f)        │
│  3. Verify Host Network Bindings (netstat / ss -tulpn)    │
│  4. Validate Environment Variables & Secret Tokens         │
└────────────────────────────────────────────────────────────┘

Diagnostic Command Sequence

Run the following diagnostic script to isolate kernel, container, and memory bottlenecks:

#!/usr/bin/env bash
set -euo pipefail

echo "=== System Kernel & Driver Status ==="
uname -r
nvidia-smi || echo "NVIDIA Driver missing or unloaded"

echo "=== Container Health & Port Allocation ==="
docker ps --format "table {{.Names}}	{{.Status}}	{{.Ports}}"

echo "=== Memory & Swap Diagnostics ==="
free -h
swapon --show

echo "=== Systemd Service Logs ==="
journalctl -u docker -n 50 --no-pager

Comprehensive Failure Modes & Solution Matrix

Error Code / Log Entry	Underlying Root Cause	Verification Command	Resolution Procedure
CUDA_ERROR_OUT_OF_MEMORY	Context window exceeds allocated VRAM	nvidia-smi	Enable 4-bit quantization (AWQ/GPTQ) or reduce --max-model-len.
docker: unknown option --gpus	Missing NVIDIA Container Toolkit hook	docker info | grep Runtime	Run sudo nvidia-ctk runtime configure and restart Docker.
502 Bad Gateway	Reverse proxy routed before backend ready	curl -Iv http://localhost:8000	Add container dependency healthchecks in docker-compose.yml.
Permission denied (publickey)	Incorrect SSH or token permission	ls -la ~/.ssh/	Fix file permissions: chmod 600 ~/.ssh/id_rsa.

---

Step-by-Step Verification Procedure

1. Verify Backend Availability: Ensure the backend container process is listening on the expected loopback interface using ss -tulpn | grep <port>.
2. Validate Payload Responses: Send a synthetic request payload using curl with verbose output -v to confirm HTTP 200 OK status.
3. Monitor System Telemetry: Watch memory and CPU counters in real time with htop or docker stats during peak request load.