How to Fix 'NVML: Driver/library version mismatch' in Docker Without Rebooting Linux

Overview

After unattended system updates (unattended-upgrades or apt upgrade), executing nvidia-smi or launching GPU Docker containers frequently fails with:

Failed to initialize NVML: Driver/library version mismatch

This occurs when the Linux kernel module version differs from the updated user-space libnvidia-ml.so library in RAM. This guide demonstrates how to reload NVIDIA kernel modules without rebooting your host server.

---

Resolution Workflow (No Reboot Required)

Step 1: Unload Dependent GPU Containers

Stop all active GPU containers:

docker stop $(docker ps -q --filter ancestor=nvidia/cuda)

Step 2: Unload Kernel Modules

Unload NVIDIA kernel modules in reverse order:

sudo rmmod nvidia_uvm
sudo rmmod nvidia_drm
sudo rmmod nvidia_modeset
sudo rmmod nvidia

Step 3: Reload NVIDIA Modules

Reload the newly updated driver modules:

sudo modprobe nvidia
sudo systemctl restart docker
nvidia-smi

Advanced Kernel Module Unloading Protocol

When kernel-space NVIDIA drivers update during background package maintenance (unattended-upgrades), running user-space processes still hold file locks on /dev/nvidia* devices. Attempting to run nvidia-smi results in the classic Driver/library version mismatch error.

Step 1: Identify and Terminate Active Process Handles

Use lsof or fuser to locate processes accessing NVIDIA character devices:

sudo lsof /dev/nvidia*

Terminate all identified container processes and background worker threads:

sudo fuser -v -k /dev/nvidia*

Step 2: Unload Module Dependencies in Exact Order

Kernel modules must be unloaded in reverse order of dependency:

# Disable nvidia-persistenced daemon
sudo systemctl stop nvidia-persistenced

# Unload dependent modules
sudo rmmod nvidia_uvm
sudo rmmod nvidia_drm
sudo rmmod nvidia_modeset
sudo rmmod nvidia

Step 3: Reload Driver & Verify DKMS Registration

Reload the kernel module matching the updated user-space library:

sudo modprobe nvidia
sudo systemctl start nvidia-persistenced
sudo systemctl restart docker

Execute verification test:

nvidia-smi

Deep Root Cause Analysis & Kernel Driver Interaction

The Failed to initialize NVML: Driver/library version mismatch error occurs when the Linux kernel’s NVIDIA kernel modules (nvidia.ko, nvidia_uvm.ko, nvidia_modeset.ko) remain running at version $V_1$ in kernel space, while host package updates or container driver mounts inject user-space CUDA and NVML libraries (libnvidia-ml.so, libcuda.so) compiled for version $V_2$.

Because NVML relies on IOCTL system calls to interact directly with /dev/nvidiactl and /dev/nvidia0, any discrepancy between the kernel driver interface protocol version and the shared object library version triggers immediate initialization rejection.

+-----------------------------------------------------------------------+
|                        User-Space Container                           |
|  +-----------------------------------------------------------------+  |
|  | PyTorch / vLLM / CUDA Runtime -> libnvidia-ml.so.1 (v550.54)     |  |
|  +-----------------------------------------------------------------+  |
+-----------------------------------||----------------------------------+
                                    || IOCTL (Version Mismatch Conflict!)
+-----------------------------------\/----------------------------------+
|                           Linux Kernel Space                          |
|  +-----------------------------------------------------------------+  |
|  | Loaded Kernel Driver Module: nvidia.ko (v535.129 loaded in RAM)  |  |
|  +-----------------------------------------------------------------+  |
+-----------------------------------------------------------------------+

Advanced Diagnostics via Sysfs & Kernel Buffer

Before executing daemon restarts, inspect kernel ring buffers and loaded kernel module states:

# Verify currently loaded kernel module version vs installed apt package version
cat /proc/driver/nvidia/version
dpkg -l | grep -i nvidia-driver

# Check systemd journal for module lock events
journalctl -u nvidia-persistenced -u docker --since "1 hour ago" --no-pager

If /proc/driver/nvidia/version outputs Driver Version 535.129.03 while dpkg displays 550.54.14, system packages were upgraded without unloading stale kernel modules.

Automated Zero-Downtime Module Reload Script

When container nodes cannot afford full system reboots, execute this defensive kernel module cycle script to safely tear down GPU file handles and re-initialize NVML:

#!/usr/bin/env bash
set -euo pipefail

echo "[*] Stopping container engine and dependent GPU daemons..."
systemctl stop docker containerd nvidia-persistenced || true

echo "[*] Unbinding active GPU processes..."
fuser -v /dev/nvidia* || true
pkill -9 -f "nvidia" || true

echo "[*] Unloading NVIDIA kernel modules in dependency order..."
modprobe -r nvidia_uvm
modprobe -r nvidia_drm
modprobe -r nvidia_modeset
modprobe -r nvidia

echo "[*] Reloading updated NVIDIA kernel modules..."
modprobe nvidia
modprobe nvidia_uvm
modprobe nvidia_modeset

echo "[*] Re-enabling persistence mode..."
nvidia-smi -pm 1

echo "[*] Restarting container engine..."
systemctl start containerd docker nvidia-persistenced

echo "[+] Verification:"
nvidia-smi
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

Production Docker Daemon Configuration Best Practices

Ensure /etc/docker/daemon.json contains explicit Cgroup configuration and sets no-cgroups = false to guarantee systemd cgroup v2 integration does not unmount GPU device paths under load:

{
  "runtimes": {
    "nvidia": {
      "args": [],
      "path": "nvidia-container-runtime"
    }
  },
  "default-runtime": "nvidia",
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "50m",
    "max-file": "3"
  }
}

After modifying daemon.json, validate syntax and trigger a daemon reload:

sudo systemctl daemon-reload
sudo systemctl restart docker

Systemd Persistence Daemon Service File Setup

Ensure vidia-persistenced is running as a systemd service to keep the NVIDIA driver loaded when no active CUDA applications or Docker containers are executing:

`ini [Unit] Description=NVIDIA Persistence Daemon Wants=syslog.target

[Service] Type=forking ExecStart=/usr/bin/nvidia-persistenced —user root ExecStopPost=/bin/rm -rf /var/run/nvidia-persistenced

[Install] WantedBy=multi-user.target `