Switch to Open NVIDIA Kernel Modules on OpenShift

💡 Quick Answer: Set driver.kernelModuleType: open in the ClusterPolicy, reboot GPU worker nodes to clear proprietary modules from kernel memory, then let the GPU Operator rebuild driver pods with open kernel modules.

The NVIDIA Open Kernel Module is required for DMA-BUF GPUDirect RDMA and GPUDirect Storage. This recipe walks through the migration from proprietary to open modules with minimal disruption.

Why Open Modules Matter: DMA-BUF vs nvidia-peermem

Open kernel modules also unlock DMA-BUF as the GPUDirect RDMA transport, replacing the out-of-tree nvidia-peermem module:

# ❌ Proprietary stack
legacy:
  kernel_modules: "Proprietary .ko (nvidia.ko, nvidia-modeset.ko)"
  gpudirect_rdma: "nvidia-peermem (out-of-tree module)"
  coupling: "Tight — kernel update breaks GPU driver"
  upgrade_risk: "High — driver rebuild per kernel version"

# ✅ Open kernel modules stack
current:
  kernel_modules: "Open kernel modules (in-tree compatible)"
  gpudirect_rdma: "DMA-BUF (upstream kernel subsystem, >= 6.x)"
  coupling: "Loose — kernel and driver independent"
  upgrade_risk: "Low — standard kernel interfaces"

Every kernel update to the proprietary stack risks rebuilding two out-of-tree modules (nvidia.ko and nvidia-peermem). With open modules and DMA-BUF, the RDMA path uses an in-tree kernel subsystem that the kernel itself maintains — only GPU userspace compatibility needs re-verifying after an upgrade.

Verify DMA-BUF is actually in use instead of nvidia-peermem after migrating:

# Should return empty — DMA-BUF replaces nvidia-peermem
kubectl exec -it gpu-pod -- lsmod | grep nvidia_peermem

# Kernel must be >= 6.x for DMA-BUF
kubectl exec -it gpu-pod -- uname -r

DMA-BUF requires kernel >= 6.x — RHEL 8 and older kernels don’t support it, so proprietary nvidia-peermem remains necessary there.

Before You Start

Verify your GPUs support open kernel modules (Turing architecture or newer):

nvidia-smi --query-gpu=gpu_name,compute_cap --format=csv

Compute capability 7.5+ (Turing) is required.

Step 1 — Check Current Module Type

# Check what the Operator is currently using
oc get clusterpolicy gpu-cluster-policy -o jsonpath='{.spec.driver.kernelModuleType}'

# Check what modules are loaded on a node
oc debug node/<gpu-node>
chroot /host
modinfo nvidia | grep -E "license|filename"

Proprietary modules show license: NVIDIA. Open modules show license: Dual MIT/GPL.

Step 2 — Update the ClusterPolicy

oc edit clusterpolicy gpu-cluster-policy

spec:
  driver:
    kernelModuleType: open

At this point the driver pods will attempt to restart, but proprietary modules may still be loaded in kernel memory on the host.

Step 3 — Cordon and Drain GPU Nodes

Process one node at a time to maintain cluster availability:

# Cordon the node to prevent scheduling
oc adm cordon <gpu-node>

# Drain workloads
oc adm drain <gpu-node> --ignore-daemonsets --delete-emptydir-data

Step 4 — Reboot the Node

Reboot clears all in-memory kernel modules:

oc debug node/<gpu-node>
chroot /host
systemctl reboot

Step 5 — Verify Open Modules After Reboot

oc debug node/<gpu-node>
chroot /host

# Verify open kernel module loaded
modinfo nvidia | grep license
# Expected: license: Dual MIT/GPL

# Verify all NVIDIA modules are on disk
find /lib/modules/$(uname -r) -name "nvidia*.ko" | head -10

# modinfo should work for all modules
modinfo nvidia_fs
modinfo nvidia_uvm

Step 6 — Uncordon the Node

oc adm uncordon <gpu-node>

Step 7 — Repeat for All GPU Nodes

Repeat Steps 3–6 for each GPU worker node in the cluster.

Step 8 — Verify Cluster-Wide Status

# All driver pods should be Running
oc get pods -n gpu-operator -l app=nvidia-driver-daemonset

# Check driver container logs for open module confirmation
oc logs -n gpu-operator ds/nvidia-driver-daemonset -c nvidia-driver-ctr | grep kernel_module_type
# Expected: kernel_module_type=open

Rollback

If you need to revert:

oc edit clusterpolicy gpu-cluster-policy

spec:
  driver:
    kernelModuleType: proprietary

Reboot nodes and restart driver pods. Note that DMA-BUF and GDS will no longer be available.

Why This Matters

Open kernel modules provide full on-disk module management, enable DMA-BUF and GPUDirect Storage, and align with NVIDIA’s recommended configuration for modern GPU Operator deployments.