Compare NCCL Intra-Node vs Inter-Node Performance

💡 Quick Answer: Run the same all_reduce_perf profile in single-node and dual-node scenarios, then compare bandwidth ratios to detect network bottlenecks.

This comparison shows whether your network fabric is limiting distributed workloads.

Standard Test Profile

all_reduce_perf -b 8 -e 1G -f 2 -g 1

Test Design

• Intra-node: two or more GPUs on one node
• Inter-node: same GPU count split across two nodes

Analysis

• Record top algbw for both scenarios
• Compute inter/intra ratio
• Investigate large drops with topology and network checks

Target Outcome

Clear and documented baseline for expected communication penalty when crossing nodes.