Fixing PyTorch Docker Bus Errors and Shared Memory Limits

When training deep learning models (PyTorch, LLMs, Computer Vision) inside Docker containers, one of the most frustrating failures is an abrupt crash with a SIGBUS (Bus Error) signal or unexpected CUDA memory errors.

This issue rarely stems from insufficient host RAM or bad GPU hardware. Instead, it is caused by a historical default in Docker: the 64 MB size limit on POSIX shared memory (/dev/shm).

This technical guide explains PyTorch’s inter-process communication (IPC) architecture, analyzes the root cause of SIGBUS crashes, and outlines optimal production configurations.

---

1. Why PyTorch Relies on POSIX Shared Memory (/dev/shm)

In PyTorch, parallel data loading (torch.utils.data.DataLoader) relies on multiprocessing when num_workers > 0.

[ PyTorch Parent Process ]
      │
      ├── Worker 1 ──► [ Loads Batch 1 ] ──┐
      ├── Worker 2 ──► [ Loads Batch 2 ] ──┼──► Writes to /dev/shm (Shared Memory)
      └── Worker 3 ──► [ Loads Batch 3 ] ──┘         │
                                                       ▼
                                          [ Main Process Reads Zero-Copy ]
                                                       │
                                                       ▼
                                          [ Transfers to VRAM (GPU) ]

To avoid duplicating Tensors in system RAM (which severely degrades I/O throughput), PyTorch uses a Zero-Copy mechanism backed by POSIX shared memory file descriptors (shm_open, mmap).

Docker’s Default Constraint

By default, Docker allocates exactly 64 MB to the /dev/shm partition in RAM (tmpfs), regardless of whether your host server has 16 GB or 512 GB of physical RAM.

As soon as your PyTorch workers load a batch of images or tokens exceeding 64 MB in total:

1. The /dev/shm partition becomes 100% full.
2. The mmap system call fails silently.
3. The Linux kernel sends a SIGBUS (Signal 7 - Bus Error) signal, killing the container process without a Python stack trace.

---

2. Symptoms and Diagnostics

Symptom 1: Silent Crash with SIGBUS

Your PyTorch script terminates abruptly mid-epoch:

ERROR: Unexpected bus error encountered in worker. This may be caused by insufficient shared memory (shm).
[Container exited with code 135]

(Exit code 135 = 128 + Signal 7 SIGBUS).

Symptom 2: Inspecting a Running Container

Verify the size of /dev/shm inside the container:

docker exec -it <container_id> df -h /dev/shm

Typical output during failure:

Filesystem      Size  Used Avail Use% Mounted on
shm              64M   64M     0 100% /dev/shm

---

3. Solutions and Best Practices

There are two primary ways to address shared memory constraints in Docker.

Option A: Increase Shared Memory Size (--shm-size) — Recommended for Production

The safest method is allocating an explicit size to /dev/shm at container startup.

1. Docker CLI

Allocate 16 GB of shared memory:

docker run --rm --device nvidia.com/gpu=all \
  --shm-size=16g \
  pytorch/pytorch:2.2.0-cuda12.1-cudnn8-runtime \
  python3 train.py

2. Docker Compose (shm_size)

In your docker-compose.yml:

version: "3.8"

services:
  pytorch-trainer:
    image: pytorch/pytorch:2.2.0-cuda12.1-cudnn8-runtime
    shm_size: "16gb"
    devices:
      - nvidia.com/gpu=all
    volumes:
      - ./dataset:/workspace/dataset
    command: python3 train.py

---

Option B: Use Host IPC Namespace (--ipc=host)

This option disables IPC namespace isolation, allowing the container to share the host’s /dev/shm partition directly.

1. Docker CLI

docker run --rm --device nvidia.com/gpu=all \
  --ipc=host \
  pytorch/pytorch:2.2.0-cuda12.1-cudnn8-runtime \
  python3 train.py

2. Docker Compose (ipc: host)

version: "3.8"

services:
  pytorch-trainer:
    image: pytorch/pytorch:2.2.0-cuda12.1-cudnn8-runtime
    ipc: host
    devices:
      - nvidia.com/gpu=all
    command: python3 train.py

Security Warning: --ipc=host removes IPC isolation. Containers running with --ipc=host can inspect or modify shared memory segments of other host processes. Prefer --shm-size in multi-tenant environments.

---

4. In-Code PyTorch Workaround

If you cannot modify Docker launch parameters (e.g., in restricted CI/CD runners), force PyTorch to use disk-backed temporary files instead of /dev/shm.

At the top of your Python script:

import torch

# Force PyTorch to use filesystem backings instead of POSIX shared memory
torch.multiprocessing.set_sharing_strategy('file_system')

Performance Trade-off

• Pros: Prevents SIGBUS crashes even on default 64 MB /dev/shm.
• Cons: Increases disk I/O overhead (can reduce training throughput by 15% to 40% if not running on high-speed NVMe storage).

---

5. Decision Matrix

Criterion	--shm-size=Xg	--ipc=host	set_sharing_strategy('file_system')
Multi-tenant Security	High (Isolated)	Low (Shared with host)	High
I/O Performance	Maximum (RAM Zero-copy)	Maximum (RAM Zero-copy)	Reduced (Disk I/O)
Flexibility	Set at launch	Uses full host RAM	Programmatically set
Ideal Use Case	Production / Kubernetes	Dedicated dev boxes	Restricted CI/CD pipelines

---

Conclusion

PyTorch SIGBUS errors in Docker are almost universally caused by the default 64 MB limit on /dev/shm. To ensure robust training pipelines:

1. Always set shm_size: "16gb" (or 30–50% of host RAM) in Docker Compose configurations.
2. Avoid --ipc=host on multi-tenant production servers.
3. Validate shared memory availability using df -h /dev/shm during MLOps pipeline checks.