Switch GPUDirect RDMA from nvidia-peermem to DMA-BUF

💡 Quick Answer: Do not set driver.rdma.enabled=true — that activates the legacy nvidia-peermem path. Instead, set driver.kernelModuleType=open and leave RDMA disabled to use the recommended DMA-BUF GPUDirect RDMA transport.

NVIDIA recommends DMA-BUF over the legacy nvidia-peermem kernel module for GPUDirect RDMA. DMA-BUF avoids a separate kernel module and is more future-proof.

Prerequisites Comparison

Step 1 — Verify Prerequisites

# Kernel version must be 5.12+
uname -r

# Check GPU architecture
nvidia-smi --query-gpu=gpu_name,compute_cap --format=csv

# Verify current module state
lsmod | grep peermem

Step 2 — Install GPU Operator for DMA-BUF

For new installations, simply omit driver.rdma.enabled=true:

# With Network Operator managing NIC drivers
helm install --wait --generate-name \
  -n gpu-operator --create-namespace \
  nvidia/gpu-operator \
  --version=v25.10.1

# With host-installed MOFED
helm install --wait --generate-name \
  -n gpu-operator --create-namespace \
  nvidia/gpu-operator \
  --version=v25.10.1 \
  --set driver.rdma.useHostMofed=true

Step 3 — Migrate Existing Installation

If you previously had driver.rdma.enabled=true, update the ClusterPolicy:

oc edit clusterpolicy gpu-cluster-policy

spec:
  driver:
    kernelModuleType: open
    rdma:
      enabled: false    # Disables legacy nvidia-peermem

Restart driver pods:

oc delete pod -n gpu-operator -l app=nvidia-driver-daemonset

Step 4 — Verify DMA-BUF is Active

Confirm nvidia-peermem-ctr container is absent:

kubectl get ds -n gpu-operator nvidia-driver-daemonset -o yaml | grep -i peermem
# Expected: no output

Check node annotations:

oc get nodes -o json | jq '.items[].metadata.annotations["nvidia.com/gpudirect-dmabuf"]'

Step 5 — Validate with NCCL

NCCL_DEBUG=INFO NCCL_IB_HCA=mlx5_0 NCCL_NET_GDR_LEVEL=5 all_reduce_test

Look for GPUDirect RDMA DMA-BUF enabled and confirm no using peer memory driver fallback.

Why This Matters

DMA-BUF is the modern, NVIDIA-recommended path that eliminates the nvidia-peermem kernel module dependency, reduces kernel version incompatibilities, and provides better long-term support.