Automating Bare-Metal GPU Server Setup: Ansible Playbook for Drivers, Docker, and CUDA 12

Executive Summary

Manual setup of bare-metal GPU hosts involves complex steps: installing kernel headers, adding signed repository keys, configuring nvidia-container-toolkit, and restarting services. This Ansible playbook automates the entire provisioning process across fresh Ubuntu 22.04/24.04 nodes in under 5 minutes.

---

Ansible Playbook Specification

Create playbook_gpu_provision.yml:

---
- hosts: gpu_servers
  become: true
  tasks:
    - name: Install system dependencies
      apt:
        name:
          - curl
          - gnupg
          - linux-headers-generic
        state: present
        update_cache: yes

    - name: Install NVIDIA Driver 535
      apt:
        name: nvidia-driver-535
        state: present

    - name: Install NVIDIA Container Toolkit
      apt:
        name: nvidia-container-toolkit
        state: present

    - name: Configure Docker daemon for NVIDIA runtime
      command: nvidia-ctk runtime configure --runtime=docker
      notify: Restart Docker

  handlers:
    - name: Restart Docker
      service:
        name: docker
        state: restarted

Idempotent Systemd Verification Role

Add verification steps to the playbook to confirm GPU persistence mode and DKMS module loading:

    - name: Enable NVIDIA Persistence Daemon
      systemd:
        name: nvidia-persistenced
        enabled: yes
        state: started

    - name: Test Docker GPU Passthrough Execution
      command: docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
      register: gpu_test_output

    - name: Display GPU Verification Output
      debug:
        var: gpu_test_output.stdout_lines

Infrastructure-as-Code GPU Provisioning Architecture

Provisioning bare-metal GPU clusters for sovereign AI workloads manually leads to driver version drift, missing kernel headers, and broken CUDA symlinks.

This Ansible playbook automates clean OS preparation, kernel header installation, official NVIDIA repository registration, CUDA toolkit installation, persistence daemon enabling, and nvidia-container-toolkit daemon integration across $N$ GPU host nodes in minutes.

+-----------------------------------------------------------------------+
|                      Ansible Control Node                             |
|  +-----------------------------------------------------------------+  |
|  | Playbook execution -> SSH (Port 22) -> Multi-Node Inventory     |  |
|  +-----------------------------------------------------------------+  |
+-------------------+-----------------------------------+---------------+
                    |                                   |
                    \/                                  \/
+-----------------------------------+   +-------------------------------+
|         GPU Host Node 01          |   |        GPU Host Node 02       |
|  +-----------------------------+  |   |  +-------------------------+  |
|  | Driver 550 + Container Runtime| |   |  | Driver 550 + Runtime    |  |
|  +-----------------------------+  |   |  +-------------------------+  |
+-----------------------------------+   +-------------------------------+

Production Ansible Playbook (gpu-provision.yml)

---
- name: Provision Bare-Metal GPU Server Nodes
  hosts: gpu_nodes
  become: true
  vars:
    nvidia_driver_version: "550"
    cuda_version: "12-4"

  tasks:
    - name: Update apt cache and install core build toolchain
      apt:
        name:
          - build-essential
          - linux-headers-{{ ansible_kernel }}
          - dkms
          - curl
          - wget
          - jq
        state: present
        update_cache: yes

    - name: Disable Nouveau open-source kernel module
      copy:
        dest: /etc/modprobe.d/blacklist-nouveau.conf
        content: |
          blacklist nouveau
          options nouveau modeset=0
        mode: '0644'

    - name: Regenerate initramfs
      command: update-initramfs -u

    - name: Download NVIDIA CUDA Keyring package
      get_url:
        url: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
        dest: /tmp/cuda-keyring.deb

    - name: Install CUDA Keyring
      apt:
        deb: /tmp/cuda-keyring.deb

    - name: Install NVIDIA Driver and CUDA Toolkit
      apt:
        name:
          - "cuda-drivers-{{ nvidia_driver_version }}"
          - "cuda-toolkit-{{ cuda_version }}"
        state: present
        update_cache: yes

    - name: Install NVIDIA Container Toolkit
      apt:
        name: nvidia-container-toolkit
        state: present

    - name: Configure Docker daemon runtime
      command: nvidia-ctk runtime configure --runtime=docker

    - name: Enable and start nvidia-persistenced service
      systemd:
        name: nvidia-persistenced
        enabled: yes
        state: started

    - name: Restart Docker engine
      systemd:
        name: docker
        state: restarted

Inventory File & Automated Execution Command

Create hosts.ini:

[gpu_nodes]
192.168.10.101 ansible_user=root
192.168.10.102 ansible_user=root
192.168.10.103 ansible_user=root

Execute rollout:

ansible-playbook -i hosts.ini gpu-provision.yml