Installing NVIDIA GPU Operator on Lightweight K3s Kubernetes Clusters for AI Pods

Executive Summary

K3s is a lightweight, production-ready Kubernetes distribution ideal for homelabs and edge servers. To run GPU workloads in K3s without manually configuring host drivers on every node, the NVIDIA GPU Operator automates driver installation, container runtime hook registration, and GPU device metrics collection.

This blueprint provides the complete Helm and manifest workflow for deploying the NVIDIA GPU Operator on K3s.

---

Step 1: K3s Cluster Configuration

Install K3s with containerd default runtime:

curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644

---

Step 2: Install NVIDIA GPU Operator via Helm

Add NVIDIA Helm repository:

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

helm install --wait --generate-name   -n gpu-operator --create-namespace   nvidia/gpu-operator   --set toolkit.enabled=true

---

Step 3: Deploy GPU Test Pod

Create gpu-test.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: cuda-vector-test
spec:
  restartPolicy: OnFailure
  containers:
    - name: cuda-test
      image: nvidia/cuda:12.4.1-base-ubuntu22.04
      command: ["nvidia-smi"]
      resources:
        limits:
          nvidia.com/gpu: 1

Apply manifest:

kubectl apply -f gpu-test.yaml
kubectl logs pod/cuda-vector-test

Homelab K3s GPU Operator Architecture

Running Kubernetes workloads on homelab edge hardware requires balancing minimal resource footprint (K3s) with fully automated GPU lifecycle management.

The NVIDIA GPU Operator uses Kubernetes custom resource definitions (CRDs) to automatically build, install, and manage host drivers, container runtimes (nvidia-container-toolkit), CUDA drivers, and Prometheus DCGM metrics exporters without altering the host OS kernel image directly.

+-----------------------------------------------------------------------+
|                    K3s Node (Lightweight Control)                     |
|  +-----------------------------------------------------------------+  |
|  | K3s Agent / Containerd Runtime (Default: runc -> nvidia-runtime) |  |
|  +-----------------------------------------------------------------+  |
|                                  | API Control                        |
|  +-------------------------------+---------------------------------+  |
|  | NVIDIA GPU Operator DaemonSets                                  |  |
|  |  +----------------+  +------------------+  +-----------------+  |  |
|  |  | Driver Installer|  | Container Toolkit|  | DCGM Exporter   |  |  |
|  |  +----------------+  +------------------+  +-----------------+  |  |
+-----------------------------------------------------------------------+

Step 1: Preparing K3s Containerd Configuration

By default, K3s manages /var/lib/rancher/k3s/agent/etc/containerd/config.toml. Configure a template override at /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl so K3s registers the nvidia runtime:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia]
  runtime_type = "io.containerd.runc.v2"
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia.options]
    BinaryName = "/usr/bin/nvidia-container-runtime"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia-experimental]
  runtime_type = "io.containerd.runc.v2"
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia-experimental.options]
    BinaryName = "/usr/bin/nvidia-container-runtime"

Restart K3s agent service to apply changes:

sudo systemctl restart k3s

Step 2: Deploying Helm Chart for NVIDIA GPU Operator

Install Helm and deploy the operator targeting the pre-configured host driver (or automated containerized drivers):

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

helm install --wait --generate-name   -n gpu-operator --create-namespace   nvidia/gpu-operator   --set driver.enabled=false   --set toolkit.enabled=true   --set toolkit.env[0].name=CONTAINERD_CONFIG   --set toolkit.env[0].value=/var/lib/rancher/k3s/agent/etc/containerd/config.toml

Step 3: Test GPU Pod Workload Manifest

Deploy a sample CUDA test workload:

apiVersion: v1
kind: Pod
metadata:
  name: cuda-test-pod
  namespace: default
spec:
  restartPolicy: OnFailure
  containers:
    - name: cuda-test
      image: nvidia/cuda:12.4.1-base-ubuntu22.04
      command: ["nvidia-smi"]
      resources:
        limits:
          nvidia.com/gpu: 1

Validate with kubectl get pod cuda-test-pod -o logs.