Profiling CUDA Applications in Docker with NVIDIA Nsight Systems

When tuning CUDA kernel execution times, Memory Copy bandwidth (H2D/D2H), or PyTorch operator latency inside Docker containers, developers frequently use NVIDIA Nsight Systems (nsys).

However, running nsys profile inside a default Docker container fails with permission errors (ERR_NVGPUCTRPERM or ptrace: Operation not permitted) because Linux kernel security restrictions block non-root performance counter sampling.

This technical guide explains why Docker blocks CUDA profiling and provides the exact security capability flags, sysctl settings, and CLI commands required to trace CUDA applications inside Docker.

---

1. Why nsys Fails in Default Docker Containers

NVIDIA Nsight Systems requires low-level kernel tracing access:

1. Linux ptrace capability: Required to inspect process memory and call stacks.
2. GPU Performance Counter Permissions (NVGPUCTRPERM): Controls access to GPU performance registers.
3. Linux Kernel perf_event_paranoid: Restricts unprivileged access to CPU performance counters.

When launching nsys profile inside a standard container, you will encounter errors like:

NVIDIA Nsight Systems: Profiling failed.
[ERR_NVGPUCTRPERM] The user does not have permission to access NVIDIA GPU Performance Counters.

---

2. Host and Container Configuration for Profiling

Step 1: Enable Host GPU Performance Counters

Allow unprivileged users to access GPU performance counters on the host:

sudo modprobe nvidia NVreg_RestrictProfilingToForcedConfigExecution=0

To make this change persistent on the host:

echo "options nvidia NVreg_RestrictProfilingToForcedConfigExecution=0" | sudo tee /etc/modprobe.d/nvidia-profiling.conf

Step 2: Configure Linux Kernel perf_event_paranoid

Set perf_event_paranoid to 1 or 0 on the host:

sudo sysctl -w kernel.perf_event_paranoid=1

---

3. Docker CLI Flags for Nsight Systems Profiling

To profile a container without granting full --privileged access, pass the specific Linux capabilities:

docker run --rm -it \
  --device nvidia.com/gpu=all \
  --cap-add=SYS_ADMIN \
  --cap-add=SYS_PTRACE \
  --security-opt seccomp=unconfined \
  -v /tmp/nsys-reports:/reports \
  nvcr.io/nvidia/pytorch:24.01-py3 \
  nsys profile \
    --trace=cuda,nvtx,osrt \
    --output=/reports/pytorch_profile_report \
    python3 train.py

Breakdown of Security Flags:

• --cap-add=SYS_ADMIN: Grants administrative privileges required for GPU performance counter sampling.
• --cap-add=SYS_PTRACE: Enables ptrace for call stack unwind sampling.
• --security-opt seccomp=unconfined: Disables seccomp syscall filtering for low-level profiler syscalls.

---

4. Profiling PyTorch CUDA Kernels with Nsight Systems & NVTX

To make Nsight Systems timelines readable, annotate your PyTorch Python code with NVTX (NVIDIA Tools Extension) ranges:

import torch
import torch.cuda.nvtx as nvtx

# Create dummy CUDA tensors
x = torch.randn(8192, 8192, device='cuda')
y = torch.randn(8192, 8192, device='cuda')

# Annotate forward pass in Nsight Systems timeline
nvtx.range_push("Matrix Multiplication Phase")
z = torch.matmul(x, y)
torch.cuda.synchronize()
nvtx.range_pop()

print("Computation Complete")

Profile Execution Command:

nsys profile \
  --trace=cuda,nvtx \
  --stats=true \
  --output=output_timeline \
  python3 script.py

---

5. Analyzing the Resulting Profile

The profiling run generates a .nsys-rep file in your mounted directory (/tmp/nsys-reports/pytorch_profile_report.nsys-rep).

1. Transfer the .nsys-rep file to your local workstation.
2. Open it in NVIDIA Nsight Systems GUI.
3. Inspect kernel execution times, GPU memory transfer bottlenecks, and NVTX markers.

---

Conclusion

Profiling CUDA in Docker containers requires unblocking specific kernel tracing barriers:

1. Host Setup: Set NVreg_RestrictProfilingToForcedConfigExecution=0.
2. Container Flags: Use --cap-add=SYS_ADMIN --cap-add=SYS_PTRACE --security-opt seccomp=unconfined.
3. NVTX Instrumentation: Wrap critical PyTorch sections with torch.cuda.nvtx.range_push() for clear visual timeline analysis.