Validate GPU and NIC Topology Before NCCL Benchmarks

💡 Quick Answer: Run nvidia-smi topo -m and lspci mapping checks first; poor physical topology often explains low NCCL bandwidth without any software bug.

Topology awareness prevents false conclusions during NCCL troubleshooting.

Commands to Run

nvidia-smi topo -m
lspci | grep -Ei 'NVIDIA|Mellanox|Ethernet|Infiniband'

What to Confirm

• GPUs used by your pod are local to the expected PCI root complex.
• High-speed NICs are attached to suitable CPU/PCI paths.
• Node hardware is homogeneous across benchmark participants.

Practical Outcome

Use topology results to define realistic performance targets for intra-node and inter-node NCCL tests.