NCCL_IB_DISABLE Environment Variable

💡 Quick Answer: Set NCCL_IB_DISABLE=1 when your cluster has InfiniBand hardware detected but you want NCCL to use TCP/Ethernet instead. Common scenarios: IB drivers installed but no IB fabric, mixed IB/Ethernet nodes, or IB errors during training. Without this flag, NCCL auto-detects IB and attempts to use it — failing silently or hanging if the fabric isn’t properly configured.

The Problem

NCCL (NVIDIA Collective Communications Library) auto-detects network interfaces. When InfiniBand hardware or drivers are present — even if not connected to an IB fabric — NCCL selects IB as the transport. This causes hangs, timeouts, or unhandled system error during distributed training. The NCCL_IB_DISABLE variable forces NCCL to skip IB and use TCP sockets instead.

flowchart TB
    NCCL["NCCL Init"] -->|"Auto-detect"| CHECK{"IB hardware<br/>present?"}
    CHECK -->|"Yes (default)"| IB["Use InfiniBand<br/>⚠️ May fail if<br/>no IB fabric"]
    CHECK -->|"No"| TCP["Use TCP/Socket"]
    
    NCCL2["NCCL Init<br/>NCCL_IB_DISABLE=1"] --> FORCE_TCP["Force TCP/Socket<br/>✅ Skip IB detection"]

The Solution

Set in Kubernetes Pod

apiVersion: v1
kind: Pod
metadata:
  name: training-job
spec:
  containers:
    - name: trainer
      image: nvcr.io/nvidia/pytorch:24.07-py3
      env:
        # Disable InfiniBand — use TCP/Ethernet
        - name: NCCL_IB_DISABLE
          value: "1"
        # Specify which Ethernet interface to use
        - name: NCCL_SOCKET_IFNAME
          value: "eth0"
        # Optional: improve TCP performance
        - name: NCCL_SOCKET_NTHREADS
          value: "4"
        - name: NCCL_NSOCKS_PERTHREAD
          value: "4"

Set in PyTorchJob

apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
  name: distributed-training
spec:
  pytorchReplicaSpecs:
    Master:
      template:
        spec:
          containers:
            - name: pytorch
              env:
                - name: NCCL_IB_DISABLE
                  value: "1"
                - name: NCCL_SOCKET_IFNAME
                  value: "eth0"
    Worker:
      replicas: 3
      template:
        spec:
          containers:
            - name: pytorch
              env:
                - name: NCCL_IB_DISABLE
                  value: "1"
                - name: NCCL_SOCKET_IFNAME
                  value: "eth0"

When to Use NCCL_IB_DISABLE=1

Diagnose IB Issues

# Check if IB devices are detected
ibstat 2>/dev/null || echo "No IB tools installed"

# Check NCCL's transport selection
NCCL_DEBUG=INFO python -c "
import torch
import torch.distributed as dist
dist.init_process_group('nccl')
" 2>&1 | grep -i "ib\|socket\|net\|transport"

# Expected output with IB disabled:
# NCCL INFO NET/Socket : Using [0]eth0:10.0.0.5<0>
# (no IB/RDMA lines)

# Expected output with IB enabled:
# NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB

Performance Comparison

# Run NCCL all-reduce benchmark with IB
NCCL_IB_DISABLE=0 \
  /opt/nccl-tests/build/all_reduce_perf -b 1M -e 1G -f 2 -g 1

# Run with TCP/Socket
NCCL_IB_DISABLE=1 NCCL_SOCKET_IFNAME=eth0 \
  /opt/nccl-tests/build/all_reduce_perf -b 1M -e 1G -f 2 -g 1

# Typical results (8 GPUs across 2 nodes):
# IB HDR (200Gb/s):    ~180 GB/s bus bandwidth
# TCP 25GbE:           ~20 GB/s bus bandwidth
# TCP 100GbE:          ~80 GB/s bus bandwidth

Related NCCL Environment Variables

Common Issues

Best Practices

• Don’t disable IB if you have working InfiniBand — 10× performance difference
• Always set NCCL_SOCKET_IFNAME with IB disabled — prevents NCCL picking loopback
• Use NCCL_DEBUG=INFO to diagnose — shows exactly which transport NCCL selects
• Set in ConfigMap for consistency — all training pods get the same NCCL config
• Test with nccl-tests before training — verify bandwidth before wasting GPU hours
• Consider EFA on AWS — better than TCP, different from IB (NCCL_IB_DISABLE=1 + FI_PROVIDER=efa)

Key Takeaways

• NCCL_IB_DISABLE=1 forces NCCL to use TCP instead of InfiniBand
• Needed when IB hardware/drivers exist but no IB fabric is connected
• Common in cloud VMs where MLNX_OFED is pre-installed
• Always pair with NCCL_SOCKET_IFNAME to select the right interface
• IB is 5-10× faster than TCP — only disable when IB isn’t available
• Use NCCL_DEBUG=INFO to verify which transport NCCL actually uses