🔧 Troubleshooting
Debug and fix: CrashLoopBackOff, OOMKilled, ImagePullBackOff, pending pods, networking issues, SR-IOV VF troubleshooting, and NFS-oRDMA performance debugging.
Fix OLM ResolutionFailed in Disconnected OpenShift
Fix ConstraintsNotSatisfiable and UnhealthyCatalogSourceFound errors when an OLM Subscription references a catalog that doesn't exist in your mirrored cluster.
OpenShift OIDC Claims Mapping Troubleshooting
Fix OpenShift OIDC login failures caused by claims mapping mismatches and wrong issuer URLs. Debug the OAuth CR openID identityProvider config.
Fix Ray Data PyTorch CPU Thread Oversubscription
Fix GPU starvation from Ray Data CPU preprocessing on Kubernetes. Pin torch.set_num_threads(1) per worker to stop thread oversubscription.
NCCL Debug Subsystems for GPU Network Troubleshooting
Configure NCCL_DEBUG and NCCL_DEBUG_SUBSYS for targeted logging during multi-node GPU training. Covers INIT, NET, GRAPH subsystems, log
NCCL Network Validation Troubleshooting Checklist
Complete troubleshooting checklist for NCCL multi-node GPU bandwidth validation. Covers SR-IOV VF allocation, /dev/infiniband visibility, RoCE GID
Kubernetes Ephemeral Containers: kubectl debug Guide
Debug running and distroless pods with kubectl debug ephemeral containers: --target, --copy-to for CrashLoopBackOff, node debugging, profiles and RBAC.
Chaos Mesh Fault Injection on Kubernetes
Deploy Chaos Mesh for chaos engineering on Kubernetes. Covers PodChaos, NetworkChaos, IOChaos, StressChaos experiments, scheduling, RBAC
LitmusChaos: Chaos Engineering on Kubernetes
Run LitmusChaos experiments on Kubernetes: install ChaosCenter, pod-delete ChaosEngine, HTTP/cmd/Prometheus probes, ChaosResult verdicts vs Chaos Mesh.
OpenShift oc cp File Copy Guide
Use oc cp to copy files and directories between local machine and Pods. Covers tar-based transfer, container selection, large file handling, and comparison
OpenShift oc rsync File Transfer
Use oc rsync to copy files between local machine and Pods in OpenShift. Covers upload, download, live sync, filtering, and common patterns for debugging
Thanos Receive OOMKilled CrashLoopBackOff
Debug and fix Thanos Receive StatefulSet OOMKilled CrashLoopBackOff caused by WAL replay exceeding memory limits. Covers ArgoCD conflict resolution, liveness
Fix Thanos Receive OOMKilled in Run:ai
Troubleshoot and fix Thanos Receive OOMKilled (exit code 137) with 143+ restarts in Run:ai backend on OpenShift. Covers memory tuning, TSDB
Kubernetes 1.36 Statusz and Flagz Endpoints
Use /statusz and /flagz debug endpoints in Kubernetes 1.36 control plane components. Inspect runtime status and effective flag values without log parsing.
kubectl describe: Read Pod Events Guide
Use kubectl describe pod to read events, conditions, and container states. Diagnose scheduling failures, image pulls, crashes, and probe failures.
kubectl exec: Run Commands in Pods
Use kubectl exec to run commands inside running pods. Interactive shell, multi-container pods, debugging techniques, and security considerations.
Fix CreateContainerError in Kubernetes
Troubleshoot Kubernetes CreateContainerError with step-by-step debugging. ConfigMap mounts, Secret references, volume permissions, and container runtime issues.
kubectl debug: Advanced Pod Debugging
Use kubectl debug for ephemeral containers, node debugging, and pod copy debugging. Debug distroless images, share process namespaces, and node-level access.
Fix Untolerated Taint node-role master
Fix 'node untolerated taint node-role.kubernetes.io/master' scheduling error. Remove or tolerate control plane taints to schedule pods on master nodes.
Journald Verify Config Kubernetes Nodes
Validate journald configuration on Kubernetes nodes. Fix journal corruption, tune storage limits, configure persistence, and troubleshoot systemd-journald.
NXDOMAIN DNS Troubleshooting Kubernetes
Fix NXDOMAIN errors in Kubernetes. Debug CoreDNS failures, ndots configuration, search domain issues, and external DNS lookup problems.
oc-mirror Troubleshooting: Fix Common Errors
Fix oc-mirror errors in disconnected OpenShift: unexpected EOF writing layers, registry 401/403, disk space, v1/v2 mismatch, graph image and delta issues.
OpenShift Cluster Operator Upgrade Debug
Debug degraded cluster operators during OpenShift upgrades. Identify stuck operators, decode status conditions, and unblock stalled rollouts.
OpenShift MCP Validation Broken Rules
Validate MachineConfigPool rules before applying in OpenShift. Detect broken MachineConfigs, degraded MCPs, and implement pre-flight checks.
SELinux SSH Login Failure Troubleshoot
Fix SSH login failures caused by SELinux enforcement. Diagnose AVC denials, restore file labels, fix custom SSH ports, and resolve PAM denials.
Cilium Debug Pod Troubleshooting
Debug Kubernetes networking with Cilium debug pods and containers. cilium-dbg, netshoot, hubble observe, and endpoint connectivity troubleshooting.
Fix CUDA Out of Memory K8s Pods
Troubleshoot CUDA out of memory errors in Kubernetes GPU pods. Memory fragmentation, batch size tuning, gradient checkpointing, and resource limits.
NVIDIA GPU Operator Troubleshooting
Fix common NVIDIA GPU Operator issues on Kubernetes. Driver pod crashes, toolkit failures, device plugin not ready, and validation pod errors.
Fix etcd Leader Election Timeout
Troubleshoot etcd leader election timeouts in K8s. Disk latency, network partition, heartbeat interval, and recovery steps.
Fix Certificate Errors Kubernetes
Troubleshoot TLS certificate errors in K8s. x509 unknown authority, expired certs, cert-manager issues, and custom CA bundles.
Fix Pod cgroup Memory Errors K8s
Fix cgroup memory limit and OOM errors in Kubernetes pods. Covers cgroup v2 migration, memory.max, swap settings, and kernel tuning for stable workloads.
Fix Service Not Reachable in Kubernetes
Debug Kubernetes Service connectivity issues. Endpoint selection, kube-proxy rules, DNS resolution, and NetworkPolicy blocks.
Fix Node NotReady Status in Kubernetes
Troubleshoot Kubernetes nodes in NotReady state. Kubelet issues, disk pressure, network problems, certificate expiration, and recovery procedures.
Kubernetes OOMKilled (Exit Code 137): Debug and Fix
Fix OOMKilled (exit code 137) in Kubernetes: find the killed container, container vs node OOM, memory limits, JVM/Node.js heap sizing, leaks, and VPA.
DNS Debugging Kubernetes Guide
Debug Kubernetes DNS issues systematically. CoreDNS troubleshooting, ndots configuration, search domains, and resolving slow DNS lookups.
Network Debugging Tools Kubernetes
Debug Kubernetes networking with tcpdump, netshoot, iptables tracing, conntrack inspection, and DNS resolution testing techniques.
Fix RBAC Permission Errors K8s
Debug Kubernetes RBAC permission errors. kubectl auth can-i, impersonation testing, ClusterRole aggregation, and common permission mistakes.
Fix 502 Bad Gateway in Kubernetes Ingress
Fix 502 Bad Gateway in Kubernetes and NGINX Ingress: empty endpoints, selector/port mismatch, rollout termination races, timeouts, and protocol mismatch.
kubectl cp Copy Files to and from Pods
Copy files between local machine and Kubernetes pods with kubectl cp. Supports containers, namespaces, tar-based transfer, and common troubleshooting.
kubectl logs View Pod Logs Guide
View and stream Kubernetes pod logs with kubectl logs. Multi-container pods, previous crashes, label selectors, timestamps, and log aggregation patterns.
Check Kubernetes Node Status with kubectl
Check and troubleshoot Kubernetes node status with kubectl. Node conditions (Ready, MemoryPressure, DiskPressure), NotReady debugging, and capacity monitoring.
Troubleshooting Pods with GPU Devices
Fix GPU device issues in Kubernetes pods. Troubleshoot device plugin errors, DRA claims, CUDA failures, driver mismatches.
Debug Kubernetes Pods: Complete Guide
Debug Kubernetes pods with kubectl debug, ephemeral containers, and netshoot. Troubleshoot distroless images, network issues, and crashed pods step by step.
Kubernetes Troubleshooting Flowchart
Systematic Kubernetes troubleshooting guide with flowcharts. Debug pods, services, networking, storage, and node issues step by step with kubectl commands.
kubectl exec: Run Commands Inside K8s Pods
Use kubectl exec to run commands inside Kubernetes pods. Covers interactive sessions, multi-container pods, and ephemeral container debugging.
Fix API Server Timeout and Overload
Debug kubectl timeouts, API server overload, and connection refused errors. Covers etcd latency, webhook timeouts, and rate limiting.
CoreDNS Troubleshooting: Fix Kubernetes DNS
Fix CoreDNS resolution failures in Kubernetes: NXDOMAIN, SERVFAIL, i/o timeouts, loop detected, ndots latency, and NetworkPolicy blocking port 53.
How to Fix CrashLoopBackOff in Kubernetes
Fix CrashLoopBackOff in Kubernetes with step-by-step troubleshooting. Debug OOMKilled, failed probes, missing configs, and image errors causing pod crash loops.
Fix etcd High Latency and Slow API Server
Debug etcd performance issues causing slow kubectl responses and API server timeouts. Covers disk I/O, compaction, defragmentation, and leader elections.
Fix fio libaio Silent Exit on OpenShift crun
Debug fio instantly exiting with no output on crun-based OpenShift nodes. The root cause is seccomp blocking libaio syscalls — fix with psync or unconfined.
Fix ImagePullBackOff and ErrImagePull in Kubernetes
Fix ImagePullBackOff, ErrImagePull and Back-off pulling image: wrong tags, pull secrets, Docker Hub rate limits, registry CA and network errors.
Fix Kubernetes Job Failures and Retries
Debug Kubernetes Jobs stuck in backoff or hitting retry limits. Covers backoffLimit, activeDeadlineSeconds, and CronJob overlap.
Fix Kubelet NotReady and Node Pressure Issues
Debug kubelet NotReady status, node pressure conditions, and eviction issues. Covers disk pressure, memory pressure, PID pressure, and network not ready.
Kubernetes Debugging Toolkit and Commands
Essential kubectl debugging commands and tools for Kubernetes troubleshooting. Covers ephemeral containers, debug pods, network debugging, and log analysis.
crun vs runc: Container Runtimes in OpenShift
Understand why pods behave differently on GPU vs CPU nodes in OpenShift. Compare crun and runc container runtimes, seccomp profiles, and syscall filtering.
Fix Pod Stuck in Pending State
Debug pods stuck in Pending status. Covers insufficient resources, node affinity mismatches, taint/toleration issues, and PVC binding failures.
Fix Podman TLS x509 Behind Corporate Proxy
Resolve podman pull x509 certificate signed by unknown authority errors caused by corporate TLS-intercepting proxies. Extract and install the proxy CA.
Fix PVC Stuck in Pending State
Debug PersistentVolumeClaims stuck in Pending status. Covers storage class issues, provisioner failures, capacity problems, and access mode mismatches.
Fix Service Mesh Sidecar Injection Failures
Debug Istio and Envoy sidecar injection issues. Covers missing sidecars, port conflicts, init container failures, and mTLS connection errors.
OpenShift oc debug Mount Limitation
Why NFS and filesystem mounts via oc debug node disappear after the debug pod exits. Understand the container namespace isolation and use MachineConfig instead.
Fix the Kubernetes ConfigMap Too Large Error
Resolve the 1MB ConfigMap size limit error. Split configs, use Secrets for binary data, mount volumes, or use external stores.
Debug CRI-O Container Runtime Errors
Troubleshoot CRI-O issues on OpenShift nodes. Fix image pull failures, container start errors, storage driver problems, and CNI networking plugin failures.
Debug Degraded MachineConfigPool Nodes
Fix nodes stuck Degraded after MachineConfig updates. Check MCD logs, on-disk validation, and recovery for degraded workers.
Kubernetes Pod Evicted: Causes, Debugging, Prevention
Why Kubernetes evicts pods: MemoryPressure, DiskPressure, ephemeral storage, preemption, NoExecute taints and drains. Diagnose, clean up, prevent.
Debug DNS Resolution Failures in Pods
Troubleshoot pods unable to resolve DNS names. Check CoreDNS health, ndots configuration, search domains, and NetworkPolicies blocking UDP port 53 DNS traffic.
Debug etcd Performance Issues in Kubernetes
Diagnose slow etcd causing API latency and leader election storms. Check disk IOPS, compaction, defrag, and network latency.
Fix Expired Certificates in Kubernetes
Renew expired certificates causing API server failures and kubelet disconnections. Manual and automatic renewal for kubeadm and OpenShift.
Fix OpenShift ImageStream Import Errors
Debug ImageStream import failures in OpenShift. Resolve DNS errors, auth issues, TLS problems, and registry rate limiting.
ITMS Race Condition with Ingress Controllers
Resolve the ITMS race condition where ImageTagMirrorSet rollouts deadlock with hostNetwork ingress controllers during MCO drain.
Fix Stale MachineConfigPool Updates
Debug and resolve stale OpenShift MachineConfigPool updates. Identify blocked nodes, check MachineConfigDaemon logs, and unblock stuck MCP rollouts.
MCP Drain Blocked by PDB: Workaround
Resolve OpenShift MachineConfigPool drain failures caused by PodDisruptionBudget violations. Scale down and restore after update.
Fix Kubernetes Namespace Stuck in Terminating
Fix a Kubernetes namespace stuck in Terminating: find leftover resources, finalizers and broken APIServices, then force-remove the namespace finalizer safely.
Debug NetworkPolicy Connectivity Issues
Troubleshoot pods unable to communicate despite correct Services. Verify NetworkPolicy rules, label selectors, and default deny.
Node Drain Blocked by hostNetwork Port Conflict
Debug and fix OpenShift node drains that fail because hostNetwork pods cannot schedule replacements due to port exhaustion across the cluster.
Debug Node NotReady Status in Kubernetes
Diagnose Kubernetes nodes stuck in NotReady state. Check kubelet logs, container runtime, network, disk pressure, and certificates.
OpenShift Ingress Router Troubleshooting
Debug OpenShift HAProxy router issues: pods stuck Pending, hostPort conflicts, PDB violations during maintenance, and custom router deployment scaling problems.
Debug MachineConfigDaemon Logs
Read and interpret OpenShift MachineConfigDaemon logs to diagnose node update failures. Common error patterns, drain issues, and config application problems.
Debug OpenShift OAuth Login Failures
Troubleshoot OpenShift console and CLI login failures. Check OAuth server pods, identity provider config, and expired tokens.
Fix Stuck OLM Operator Subscriptions
Debug Operator Lifecycle Manager subscriptions stuck in pending or failed state. Resolve catalog source issues, approval policies, and CSV dependency conflicts.
PDB Allowed Disruptions Zero: Debugging
Debug PodDisruptionBudgets stuck at zero allowed disruptions. Understand minAvailable vs maxUnavailable, fix eviction failures, and plan for maintenance.
Fix PV Stuck in Terminating State
Resolve PVs and PVCs stuck in Terminating status. Remove finalizers safely, check volume detachment, and handle storage issues.
Fix ResourceQuota Exceeded Errors
Debug resource quota violations preventing pod scheduling. Understand LimitRange defaults, ResourceQuota, and namespace management.
Debug Service with No Ready Endpoints
Troubleshoot Services showing zero endpoints. Verify label selectors, readiness probes, pod status, and port configuration.
Fix Node Untolerated Taint Scheduling Errors
Fix node untolerated taint errors causing pods stuck in Pending. NoSchedule, PreferNoSchedule, NoExecute effects, and toleration syntax guide.
Fix Admission Webhook Timeout Errors
Debug admission webhook failures blocking pod creation. Identify failing webhooks, check timeouts, and set failurePolicy.
Decode and Inspect Kubernetes Docker Secrets
Decode base64-encoded dockerconfigjson secrets to verify registry credentials, troubleshoot ImagePullBackOff errors, and audit pull secret configurations.
Troubleshoot CatalogSource and OLM Issues
Debug CatalogSource failures including pod crashes, gRPC errors, stale caches, and operator install problems in OpenShift OLM environments.
SR-IOV VF Troubleshooting on Kubernetes
Diagnose and fix SR-IOV Virtual Function issues including VF creation failures, device plugin errors, RDMA problems, and network attachment failures.
Diagnose NVIDIA Memory-Only Kernel Modules
Understand why lsmod shows NVIDIA modules loaded but modinfo fails, and how the GPU Operator's proprietary driver container inserts modules without.
Fix NVIDIA Peer Memory Driver Not Detected
Diagnose and resolve the 'NVIDIA peer memory driver not detected' error when running GPU workloads with RDMA on Kubernetes and OpenShift.
Fix nvidia-fs Module Conflict on OpenShift
Diagnose and fix the 'insmod: ERROR: could not insert module nvidia-fs.ko: File exists' error when enabling GPUDirect Storage with the NVIDIA GPU Operator.
Debug NCCL Timeouts and Hangs in Kubernetes
Systematically troubleshoot NCCL runs that stall or timeout across multi-GPU and multi-node Kubernetes jobs with step-by-step diagnostic commands.
Diagnose GPU Peer-to-Peer Latency NCCL Tests
Use NCCL point-to-point and collective tests to isolate GPU peer-to-peer latency issues between GPU pairs in multi-node Kubernetes clusters.
Validate GPU & NIC Topology Before NCCL
Inspect node-level GPU, NIC, and PCI topology on Kubernetes workers to predict and explain NCCL benchmark performance before running tests.
Check Bonding and Interface Status for SR-IOV
Inspect bond membership, interface state, and link aggregation to confirm which NICs can be correctly targeted by SR-IOV network policies on Kubernetes.
Identify Mellanox NIC Models from Linux
Map interface names to PCI addresses and Mellanox model generations to build accurate SR-IOV policies and GPU networking configurations on Kubernetes.
Fix NVIDIA NIM TensorRT-LLM Initialization Errors
Diagnose and fix common NIM TensorRT-LLM executor failures including DecoderState mismatch, version incompatibilities, and engine build errors.
Fix nv-ipam 'Pool Not Found' Errors in Multus
Fix nv-ipam IPPool lookup failures in Multus by aligning SriovNetwork, NetworkAttachmentDefinition, and IPPool names and namespaces correctly.
Fix 'No Supported NIC Is Selected' in SR-IOV
Diagnose SR-IOV operator webhook rejections by validating node state, label selectors, PF eligibility, and SriovNetworkNodePolicy configuration.
Validate SR-IOV Operator Health Across Nodes
Run a full checklist to confirm SR-IOV discovery, VF creation, scheduler resources, and pod attachment on multiple nodes.
How to Troubleshoot Kubernetes Networking
Debug and resolve Kubernetes networking issues systematically. Learn to diagnose DNS problems, service connectivity, network policies, and CNI issues.
How to Debug Kubernetes Node Issues
Diagnose and troubleshoot node problems in Kubernetes clusters. Identify resource pressure, connectivity issues, and component failures.
How to Debug Pod Networking Issues
Diagnose and fix Kubernetes networking problems. Troubleshoot connectivity, DNS resolution, service discovery, and network policies with practical tools.
Debug Pod Scheduling Failures in K8s
Fix pods stuck in Pending from scheduling failures. Diagnose resource constraints, node affinity, taints, tolerations, and topology spread conflicts.
How to Run Kubernetes in Docker (kind)
Create local Kubernetes clusters using kind (Kubernetes in Docker). Set up multi-node clusters, configure networking, and test applications locally.
Kubernetes Finalizers: Remove Stuck Finalizers
What Kubernetes finalizers are and how to safely remove stuck ones with kubectl patch. Fix namespaces, PVCs, CRDs and Helm releases stuck in Terminating.