Debug NCCL Timeouts and Hangs in Kubernetes

💡 Quick Answer: Enable NCCL_DEBUG=INFO, inspect transport selection logs, verify interface configuration, and re-run with a reduced pod/node matrix to isolate the failing path.

NCCL hangs usually come from transport setup failures, network asymmetry, or inconsistent node state.

Step-by-Step

1. Enable detailed logs:

export NCCL_DEBUG=INFO

2. Check pod events and restart reasons:

kubectl describe pod <pod-name> -n <namespace>

3. Validate interface and routing inside each pod.

4. Re-run with fewer nodes/GPUs to isolate the issue.

High-Value Checks

• Same container image across all participants
• Same driver/runtime compatibility on all nodes
• No hidden policy blocking east-west traffic

Resolution Pattern

Start from a known-good single-node run, then scale one dimension at a time.