Run NCCL AllGather Benchmarks for Model Parallel Validation

💡 Quick Answer: Execute all_gather_perf -b 8 -e 1G -f 2 -g 1 to validate communication efficiency for model-parallel patterns.

All-gather performance is important for tensor-parallel inference and training pipelines.

Benchmark Command

all_gather_perf -b 8 -e 1G -f 2 -g 1

Execution Tips

• Keep pod CPU/memory limits realistic to avoid host bottlenecks.
• Use fixed node placement between runs.
• Run at least 3 iterations and compare averages.

Troubleshooting Signals

• Large variance between runs suggests noisy neighbors or unstable links.
• Sudden drops at specific sizes can indicate MTU or transport fallback issues.

Output to Store

Save logs with node names, GPU count, and NIC interface metadata for trend analysis.