Self-Hosted LLM Fine-Tuning Pipeline: Unsloth, LLoRA, and Docker Compose

Overview

Fine-tuning open-weight models (like Llama 3, Mistral, or Qwen) on proprietary domain datasets is essential for enterprise AI. However, standard Hugging Face TRL fine-tuning pipelines consume vast amounts of VRAM and memory. Unsloth accelerates fine-tuning speeds by 2-5x while cutting VRAM requirements by 80% using custom Triton kernels.

This guide provides a production Docker Compose setup for fine-tuning LLMs with Unsloth and exporting GGUF/AWQ weights.

---

Docker Compose Fine-Tuning Environment

Create /opt/unsloth-pipeline/docker-compose.yml:

version: "3.8"

services:
  unsloth_trainer:
    image: unsloth/unsloth-notebook:latest
    container_name: unsloth_trainer
    restart: "no"
    ipc: host
    environment:
      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}
    volumes:
      - ./data:/workspace/data:rw
      - ./output:/workspace/output:rw
    ports:
      - "8888:8888"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

---

Python QLoRA Fine-Tuning Script

Create /opt/unsloth-pipeline/data/train.py:

from unsloth import FastLanguageModel
import torch

max_seq_length = 2048
dtype = None # Auto detect

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/llama-3-8b-Instruct-bnb-4bit",
    max_seq_length = max_seq_length,
    dtype = dtype,
    load_in_4bit = True,
)

model = FastLanguageModel.get_peft_model(
    model,
    r = 16,
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_alpha = 16,
    lora_dropout = 0,
    bias = "none",
)

# Export Merged 16bit / GGUF model
model.save_pretrained_merged("output/llama-3-8b-custom", tokenizer, save_method = "merged_16bit")

---

Summary Checklist

• Provisioned Docker container with Unsloth Triton kernels.
• Configured 4-bit QLoRA training parameters.
• Exported merged model weights for vLLM / Ollama serving.

Fine-Tuning Optimization Architecture

Fine-tuning large language models (LLMs) like Llama 3.3 70B or Qwen 2.5 on custom enterprise data traditionally requires massive VRAM overhead.

Unsloth optimizes Low-Rank Adaptation (LoRA) and QLoRA kernel operations by rewriting PyTorch compute graphs directly into optimized Triton C-code. This reduces VRAM requirements by up to 80% while accelerating training speeds by 2x-5x compared to Hugging Face peft and TRL baselines.

+-------------------------------------------------------------------------+
|                  Unsloth QLoRA Docker Environment                       |
|  +-------------------------------------------------------------------+  |
|  | Custom Dataset (JSONL) -> Tokenizer -> Unsloth Triton Kernels     |  |
|  +-------------------------------------------------------------------+  |
|                                   | 4-bit / 8-bit Quantized Base Model  |
|                                   \/                                    |
|  +-------------------------------------------------------------------+  |
|  | NVIDIA GPU Memory (VRAM)                                          |  |
|  | [ Base Model Weights (Frozen) ] + [ Trainable LoRA Adapters ]     |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

Step 1: Docker Container Deployment (docker-compose.yml)

version: "3.8"

services:
  unsloth-trainer:
    image: unsloth/unsloth:latest
    container_name: unsloth-pipeline
    restart: "no"
    runtime: nvidia
    environment:
      - CUDA_VISIBLE_DEVICES=0
      - WANDB_DISABLED=true
    ipc: host
    volumes:
      - ./data:/workspace/data
      - ./output:/workspace/output
      - ./scripts:/workspace/scripts
    command: python3 /workspace/scripts/train.py

Step 2: Unsloth Fine-Tuning Execution Script (scripts/train.py)

from unsloth import FastLanguageModel
import torch
from datasets import load_dataset
from trl import SFTTrainer
from transformers import TrainingArguments

# 1. Load Model & Tokenizer with 4-bit Quantization
max_seq_length = 4096
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/Llama-3.2-3B-Instruct",
    max_seq_length = max_seq_length,
    load_in_4bit = True,
)

# 2. Add Target LoRA Adapters
model = FastLanguageModel.get_peft_model(
    model,
    r = 16,
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    lora_alpha = 16,
    lora_dropout = 0,
    bias = "none",
)

# 3. Load & Format Dataset
dataset = load_dataset("json", data_files={"train": "/workspace/data/dataset.jsonl"})

# 4. Configure SFTTrainer
trainer = SFTTrainer(
    model = model,
    tokenizer = tokenizer,
    train_dataset = dataset["train"],
    dataset_text_field = "text",
    max_seq_length = max_seq_length,
    args = TrainingArguments(
        per_device_train_batch_size = 2,
        gradient_accumulation_steps = 4,
        warmup_steps = 10,
        max_steps = 100,
        learning_rate = 2e-4,
        fp16 = not torch.cuda.is_bf16_supported(),
        bf16 = torch.cuda.is_bf16_supported(),
        logging_steps = 1,
        output_dir = "/workspace/output",
    ),
)

# 5. Execute Training & Export GGUF / LoRA
trainer.train()
model.save_pretrained_merged("/workspace/output/final_model", tokenizer, save_method="merged_16bit")
print("[+] Model fine-tuning and export completed successfully.")