Tune NCCL Environment Variables for RDMA and Ethernet

💡 Quick Answer: Start with NCCL_DEBUG=INFO, set NCCL_SOCKET_IFNAME to the correct data interface, and enable or disable InfiniBand explicitly using NCCL_IB_DISABLE.

Use explicit NCCL environment configuration to reduce transport ambiguity and improve repeatability.

RDMA-Oriented Profile

NCCL_DEBUG=INFO
NCCL_IB_DISABLE=0
NCCL_SOCKET_IFNAME=eth0

Ethernet-Only Profile

NCCL_DEBUG=INFO
NCCL_IB_DISABLE=1
NCCL_SOCKET_IFNAME=eth0

Validation Loop

1. Apply one profile.
2. Run all_reduce_perf and keep logs.
3. Compare bandwidth and error rates.

Best Practices

• Change one variable at a time when troubleshooting.
• Keep per-cluster baseline profiles under version control.
• Re-test after CNI, firmware, or driver upgrades.