https://kubernetes.recipes/ Production-ready Kubernetes and OpenShift recipes — deployments, networking, security, AI/GPU, storage, and troubleshooting. en-us Sun, 07 Jun 2026 13:49:20 GMT https://kubernetes.recipes/recipes/ai/disable-gds-enable-iommu-passthrough-kubernetes/ https://kubernetes.recipes/recipes/ai/disable-gds-enable-iommu-passthrough-kubernetes/ Thu, 04 Jun 2026 00:00:00 GMT AI & GPU Disable GPUDirect Storage (GDS) when not needed and configure IOMMU passthrough mode for GPU and NIC device assignment. Kernel parameters, BIOS settings, VFIO https://kubernetes.recipes/recipes/ai/gpu-operator-clusterpolicy-rdma-gds-iommu/ https://kubernetes.recipes/recipes/ai/gpu-operator-clusterpolicy-rdma-gds-iommu/ Thu, 04 Jun 2026 00:00:00 GMT AI & GPU Configure NVIDIA GPU Operator ClusterPolicy to disable RDMA and enable GPUDirect Storage (GDS). Control nvidia-peermem, nvidia-fs modules, driver https://kubernetes.recipes/recipes/ai/gpudirect-rdma-setup-verification-kubernetes/ https://kubernetes.recipes/recipes/ai/gpudirect-rdma-setup-verification-kubernetes/ Thu, 04 Jun 2026 00:00:00 GMT AI & GPU Enable and verify GPUDirect RDMA for GPU-to-NIC direct data transfer on Kubernetes. Install nvidia-peermem, configure DMA-BUF, verify RDMA paths, troubleshoot https://kubernetes.recipes/recipes/ai/iommu-kernel-parameters-gpu-nodes-kubernetes/ https://kubernetes.recipes/recipes/ai/iommu-kernel-parameters-gpu-nodes-kubernetes/ Thu, 04 Jun 2026 00:00:00 GMT AI & GPU Configure IOMMU kernel parameters for optimal GPU and RDMA performance on Kubernetes. Compare intel_iommu, amd_iommu, and iommu settings, passthrough vs off vs https://kubernetes.recipes/recipes/ai/kubeflow-mpijob-worker-ssh-gpu-training/ https://kubernetes.recipes/recipes/ai/kubeflow-mpijob-worker-ssh-gpu-training/ Thu, 04 Jun 2026 00:00:00 GMT AI & GPU Configure SSH daemon in Kubeflow MPIJob worker pods for multi-node GPU training. Covers SSHD setup in containers, host key generation, authorized keys from MPI https://kubernetes.recipes/recipes/ai/kubernetes-topology-manager-gpu-numa/ https://kubernetes.recipes/recipes/ai/kubernetes-topology-manager-gpu-numa/ Thu, 04 Jun 2026 00:00:00 GMT AI & GPU Configure Kubernetes Topology Manager to align CPU, GPU, and NIC allocations on the same NUMA node. Covers policies, kubelet config, and GPU performance tuning. https://kubernetes.recipes/recipes/ai/mpi-dns-resolution-hostfile-kubernetes/ https://kubernetes.recipes/recipes/ai/mpi-dns-resolution-hostfile-kubernetes/ Thu, 04 Jun 2026 00:00:00 GMT AI & GPU Troubleshoot MPI hostfile DNS resolution in Kubeflow MPIJob on Kubernetes. Covers headless Service creation, subdomain configuration, DNS wait loops, FQDN https://kubernetes.recipes/recipes/ai/nccl-all-reduce-perf-benchmark-multi-node/ https://kubernetes.recipes/recipes/ai/nccl-all-reduce-perf-benchmark-multi-node/ Thu, 04 Jun 2026 00:00:00 GMT AI & GPU Run and interpret NCCL all_reduce_perf benchmarks on multi-node Kubernetes GPU clusters. Understand bus bandwidth results, expected throughput for H200 NVL https://kubernetes.recipes/recipes/ai/nccl-channel-routing-transport-analysis/ https://kubernetes.recipes/recipes/ai/nccl-channel-routing-transport-analysis/ Thu, 04 Jun 2026 00:00:00 GMT AI & GPU Interpret NCCL channel logs to understand GPU communication paths on Kubernetes. Decode P2P/CUMEM, SHM/direct, NET/IB/GDRDMA transport https://kubernetes.recipes/recipes/troubleshooting/nccl-debug-subsystems-gpu-troubleshooting/ https://kubernetes.recipes/recipes/troubleshooting/nccl-debug-subsystems-gpu-troubleshooting/ Thu, 04 Jun 2026 00:00:00 GMT Troubleshooting Configure NCCL_DEBUG and NCCL_DEBUG_SUBSYS for targeted logging during multi-node GPU training. Covers INIT, NET, GRAPH subsystems, log https://kubernetes.recipes/recipes/ai/nccl-dmabuf-gpudirect-rdma-kubernetes/ https://kubernetes.recipes/recipes/ai/nccl-dmabuf-gpudirect-rdma-kubernetes/ Thu, 04 Jun 2026 00:00:00 GMT AI & GPU Enable NCCL DMA-BUF support for GPUDirect RDMA in Kubernetes GPU clusters. Covers NCCL_DMABUF_ENABLE=1, kernel requirements, nvidia-peermem vs dmabuf, GPU https://kubernetes.recipes/recipes/ai/nccl-gdr-level-tuning-pix-pxb-phb-sys/ https://kubernetes.recipes/recipes/ai/nccl-gdr-level-tuning-pix-pxb-phb-sys/ Thu, 04 Jun 2026 00:00:00 GMT AI & GPU Tune NCCL_NET_GDR_LEVEL for optimal GPUDirect RDMA performance on Kubernetes. Compare PIX, PXB, PHB, and SYS distance thresholds with PCIe topology. Benchmark https://kubernetes.recipes/recipes/ai/nccl-gpudirect-rdma-distance-pix-sys/ https://kubernetes.recipes/recipes/ai/nccl-gpudirect-rdma-distance-pix-sys/ Thu, 04 Jun 2026 00:00:00 GMT AI & GPU Understand NCCL GPU Direct RDMA distance-based enablement. When PIX mode disables GDRDMA for distant GPU-HCA pairs (distance 9 > 4) and when SYS mode enables https://kubernetes.recipes/recipes/ai/nccl-ib-hca-qps-tuning-roce/ https://kubernetes.recipes/recipes/ai/nccl-ib-hca-qps-tuning-roce/ Thu, 04 Jun 2026 00:00:00 GMT AI & GPU Configure NCCL_IB_HCA, NCCL_IB_GID_INDEX, NCCL_IB_QPS_PER_CONNECTION, and NCCL_IB_SPLIT_DATA_ON_QPS for optimal RoCE performance on Kubernetes GPU clusters. https://kubernetes.recipes/recipes/ai/nccl-network-validation-script-openshift/ https://kubernetes.recipes/recipes/ai/nccl-network-validation-script-openshift/ Thu, 04 Jun 2026 00:00:00 GMT AI & GPU Build a comprehensive NCCL network validation script for OpenShift GPU clusters with SR-IOV. Configure NCCL_IB_GID_INDEX, NCCL_NET_GDR_LEVEL=SYS, per-rank HCA https://kubernetes.recipes/recipes/troubleshooting/nccl-network-validation-troubleshooting-checklist/ https://kubernetes.recipes/recipes/troubleshooting/nccl-network-validation-troubleshooting-checklist/ Thu, 04 Jun 2026 00:00:00 GMT Troubleshooting Complete troubleshooting checklist for NCCL multi-node GPU bandwidth validation. Covers SR-IOV VF allocation, /dev/infiniband visibility, RoCE GID https://kubernetes.recipes/recipes/ai/nccl-network-validator-production-mpijob/ https://kubernetes.recipes/recipes/ai/nccl-network-validator-production-mpijob/ Thu, 04 Jun 2026 00:00:00 GMT AI & GPU Deploy a production-ready NCCL network validation framework using Kubeflow MPIJob on OpenShift. Complete validate_network.sh script https://kubernetes.recipes/recipes/ai/nccl-roce-validation-mpijob-complete-reference/ https://kubernetes.recipes/recipes/ai/nccl-roce-validation-mpijob-complete-reference/ Thu, 04 Jun 2026 00:00:00 GMT AI & GPU Complete nccl-roce-validation.yaml MPIJob reference for OpenShift GPU clusters. Full launcher environment variables, OpenMPI control plane settings, NCCL https://kubernetes.recipes/recipes/ai/nccl-roce-validation-mpijob-kubernetes/ https://kubernetes.recipes/recipes/ai/nccl-roce-validation-mpijob-kubernetes/ Thu, 04 Jun 2026 00:00:00 GMT AI & GPU Run NCCL all_reduce_perf validation tests using Kubeflow MPIJob on GPU clusters. Configure MPI launcher and workers, NCCL environment variables, test https://kubernetes.recipes/recipes/ai/nccl-shared-memory-transport-kubernetes/ https://kubernetes.recipes/recipes/ai/nccl-shared-memory-transport-kubernetes/ Thu, 04 Jun 2026 00:00:00 GMT AI & GPU Configure NCCL shared memory (SHM) transport for intra-node GPU communication on Kubernetes. Covers /dev/shm sizing with emptyDir and NVLink/PCIe P2P paths. https://kubernetes.recipes/recipes/ai/nvidia-gpu-topology-matrix-kubernetes/ https://kubernetes.recipes/recipes/ai/nvidia-gpu-topology-matrix-kubernetes/ Thu, 04 Jun 2026 00:00:00 GMT AI & GPU Read and interpret nvidia-smi topo and nvidia-device-plugin topology matrices on Kubernetes GPU nodes. Understand X, NV, SYS, NODE, PIX, PXB, PHB connection https://kubernetes.recipes/recipes/networking/nvidia-network-operator-rdma-kubernetes/ https://kubernetes.recipes/recipes/networking/nvidia-network-operator-rdma-kubernetes/ Thu, 04 Jun 2026 00:00:00 GMT Networking Deploy and configure RDMA for GPU clusters using the NVIDIA Network Operator. NicClusterPolicy setup, MLNX_OFED driver container, shared and SR-IOV RDMA device https://kubernetes.recipes/recipes/ai/nvlink-bridge-architecture-gpu-kubernetes/ https://kubernetes.recipes/recipes/ai/nvlink-bridge-architecture-gpu-kubernetes/ Thu, 04 Jun 2026 00:00:00 GMT AI & GPU Understand NVLink Bridge logical architecture in GPU servers for Kubernetes. Dual-socket PCIe Gen5 topology, NVL4 groups, GPU-NIC-NVMe placement, PCIe switch https://kubernetes.recipes/recipes/ai/openmpi-control-plane-separation-nccl-rdma/ https://kubernetes.recipes/recipes/ai/openmpi-control-plane-separation-nccl-rdma/ Thu, 04 Jun 2026 00:00:00 GMT AI & GPU Configure OpenMPI to use eth0 for MPI control traffic while NCCL uses net1 SR-IOV for data. Covers btl_tcp_if_include, pml, routed direct, plm_rsh_agent SSH https://kubernetes.recipes/recipes/networking/openshift-sriov-network-nv-ipam-gpu-fabric/ https://kubernetes.recipes/recipes/networking/openshift-sriov-network-nv-ipam-gpu-fabric/ Thu, 04 Jun 2026 00:00:00 GMT Networking Configure SriovNetwork resources on OpenShift with nv-ipam for GPU fabric IP allocation. SR-IOV Network Operator setup, Mellanox NIC resource targeting, IPAM https://kubernetes.recipes/recipes/ai/runai-gpu-scheduling-kubeflow-mpijob/ https://kubernetes.recipes/recipes/ai/runai-gpu-scheduling-kubeflow-mpijob/ Thu, 04 Jun 2026 00:00:00 GMT AI & GPU Integrate Run:ai GPU scheduler with Kubeflow MPIJob for multi-node NCCL workloads. Covers Run:ai project namespaces, GPU quota annotations, pod group https://kubernetes.recipes/recipes/networking/shared-rdma-device-plugin-kubernetes/ https://kubernetes.recipes/recipes/networking/shared-rdma-device-plugin-kubernetes/ Thu, 04 Jun 2026 00:00:00 GMT Networking Configure the RDMA shared device plugin to allow multiple pods to share RDMA-capable NICs on Kubernetes. K8s-rdma-shared-dev-plugin setup, resource https://kubernetes.recipes/recipes/networking/sriov-multus-network-attachment-gpu-rdma/ https://kubernetes.recipes/recipes/networking/sriov-multus-network-attachment-gpu-rdma/ Thu, 04 Jun 2026 00:00:00 GMT Networking Configure Multus CNI NetworkAttachmentDefinition for SR-IOV RDMA in Kubernetes GPU workloads. Covers k8s.v1.cni.cncf.io/networks annotation, IPAM https://kubernetes.recipes/recipes/storage/cloudnativepg-postgresql-operator-kubernetes/ https://kubernetes.recipes/recipes/storage/cloudnativepg-postgresql-operator-kubernetes/ Mon, 01 Jun 2026 00:00:00 GMT Storage Deploy production PostgreSQL on Kubernetes with CloudNativePG operator. Automated failover, continuous backup to S3, point-in-time recovery, connection https://kubernetes.recipes/recipes/configuration/crossplane-kubernetes-infrastructure-management/ https://kubernetes.recipes/recipes/configuration/crossplane-kubernetes-infrastructure-management/ Mon, 01 Jun 2026 00:00:00 GMT Configuration Manage cloud infrastructure as Kubernetes resources with Crossplane. Provision AWS, GCP, and Azure resources using custom resource https://kubernetes.recipes/recipes/ai/genai-perf-benchmarking-llm-inference-kubernetes/ https://kubernetes.recipes/recipes/ai/genai-perf-benchmarking-llm-inference-kubernetes/ Mon, 01 Jun 2026 00:00:00 GMT AI & GPU Benchmark LLM inference performance with NVIDIA GenAI-Perf on Kubernetes. Profile vLLM, TensorRT-LLM, and Triton endpoints with concurrency sweeps, token https://kubernetes.recipes/recipes/observability/grafana-kubernetes-monitoring-dashboards/ https://kubernetes.recipes/recipes/observability/grafana-kubernetes-monitoring-dashboards/ Mon, 01 Jun 2026 00:00:00 GMT Observability Deploy and configure Grafana dashboards for Kubernetes monitoring including dashboard 6417 for pod metrics, dashboard 315 for cluster overview, and custom https://kubernetes.recipes/recipes/helm/helm-sprig-functions-complete-reference/ https://kubernetes.recipes/recipes/helm/helm-sprig-functions-complete-reference/ Mon, 01 Jun 2026 00:00:00 GMT Helm Complete reference for Helm Sprig template functions including cat, print, join, tostring, add1, trim, quote, default, and more. Examples and common patterns https://kubernetes.recipes/recipes/autoscaling/keda-event-driven-autoscaling-kubernetes/ https://kubernetes.recipes/recipes/autoscaling/keda-event-driven-autoscaling-kubernetes/ Mon, 01 Jun 2026 00:00:00 GMT Autoscaling Deploy KEDA for event-driven autoscaling on Kubernetes. Scale deployments to zero based on queue depth, HTTP requests, cron schedules, Prometheus https://kubernetes.recipes/recipes/security/kubernetes-audit-logging-configuration/ https://kubernetes.recipes/recipes/security/kubernetes-audit-logging-configuration/ Mon, 01 Jun 2026 00:00:00 GMT Security Configure Kubernetes audit logging to track API requests. Define audit policies, capture who did what and when, send logs to backends like https://kubernetes.recipes/recipes/deployments/kubernetes-blue-green-canary-deployments/ https://kubernetes.recipes/recipes/deployments/kubernetes-blue-green-canary-deployments/ Mon, 01 Jun 2026 00:00:00 GMT Deployments Implement blue-green and canary deployment strategies on Kubernetes. Zero-downtime releases using Service label switching, traffic splitting, progressive https://kubernetes.recipes/recipes/configuration/kubernetes-cronjob-concurrencypolicy-guide/ https://kubernetes.recipes/recipes/configuration/kubernetes-cronjob-concurrencypolicy-guide/ Mon, 01 Jun 2026 00:00:00 GMT Configuration Configure Kubernetes CronJob concurrencyPolicy with Allow, Forbid, and Replace options. Control concurrent job execution, prevent overlapping runs, and handle https://kubernetes.recipes/recipes/deployments/kubernetes-daemonset-one-pod-per-node/ https://kubernetes.recipes/recipes/deployments/kubernetes-daemonset-one-pod-per-node/ Mon, 01 Jun 2026 00:00:00 GMT Deployments Deploy DaemonSets on Kubernetes to run exactly one pod per node. Configure tolerations, node selectors, affinity rules, and resource management https://kubernetes.recipes/recipes/observability/kubernetes-efk-stack-centralized-logging/ https://kubernetes.recipes/recipes/observability/kubernetes-efk-stack-centralized-logging/ Mon, 01 Jun 2026 00:00:00 GMT Observability Deploy the EFK stack (Elasticsearch, Fluentd, Kibana) on Kubernetes for centralized log collection, processing, and visualization. DaemonSet log https://kubernetes.recipes/recipes/configuration/kubernetes-envfrom-configmap-environment-variables/ https://kubernetes.recipes/recipes/configuration/kubernetes-envfrom-configmap-environment-variables/ Mon, 01 Jun 2026 00:00:00 GMT Configuration Inject all ConfigMap keys as environment variables using envFrom in Kubernetes pods. Configure configMapRef, secretRef, prefix options, and selective key https://kubernetes.recipes/recipes/troubleshooting/kubernetes-ephemeral-containers-debugging/ https://kubernetes.recipes/recipes/troubleshooting/kubernetes-ephemeral-containers-debugging/ Mon, 01 Jun 2026 00:00:00 GMT Troubleshooting Debug running pods with Kubernetes ephemeral containers. Attach debug containers without restarting pods, troubleshoot distroless images, inspect network https://kubernetes.recipes/recipes/troubleshooting/kubernetes-finalizers-explained-troubleshooting/ https://kubernetes.recipes/recipes/troubleshooting/kubernetes-finalizers-explained-troubleshooting/ Mon, 01 Jun 2026 00:00:00 GMT Troubleshooting Understand Kubernetes finalizers for resource cleanup. How finalizers block deletion, common stuck resource scenarios, manual removal https://kubernetes.recipes/recipes/deployments/kubernetes-graceful-shutdown-pod-termination/ https://kubernetes.recipes/recipes/deployments/kubernetes-graceful-shutdown-pod-termination/ Mon, 01 Jun 2026 00:00:00 GMT Deployments Implement graceful shutdown for Kubernetes pods. Configure terminationGracePeriodSeconds, preStop hooks, SIGTERM handling, connection https://kubernetes.recipes/recipes/security/kubernetes-gvisor-kata-containers-runtimeclass/ https://kubernetes.recipes/recipes/security/kubernetes-gvisor-kata-containers-runtimeclass/ Mon, 01 Jun 2026 00:00:00 GMT Security Deploy sandboxed container runtimes on Kubernetes using RuntimeClass with gVisor (runsc) and Kata Containers. Isolate untrusted workloads with kernel-level https://kubernetes.recipes/recipes/autoscaling/kubernetes-hpa-custom-metrics-prometheus-adapter/ https://kubernetes.recipes/recipes/autoscaling/kubernetes-hpa-custom-metrics-prometheus-adapter/ Mon, 01 Jun 2026 00:00:00 GMT Autoscaling Configure Kubernetes Horizontal Pod Autoscaler with custom Prometheus metrics via the Prometheus Adapter. Scale on request latency, queue depth, GPU https://kubernetes.recipes/recipes/troubleshooting/kubernetes-imagepullbackoff-troubleshooting/ https://kubernetes.recipes/recipes/troubleshooting/kubernetes-imagepullbackoff-troubleshooting/ Mon, 01 Jun 2026 00:00:00 GMT Troubleshooting Debug and fix ImagePullBackOff and ErrImagePull errors in Kubernetes. Resolve authentication failures, registry connectivity, image not found, TLS certificate https://kubernetes.recipes/recipes/networking/kubernetes-ingress-tls-cert-manager/ https://kubernetes.recipes/recipes/networking/kubernetes-ingress-tls-cert-manager/ Mon, 01 Jun 2026 00:00:00 GMT Networking Automate TLS certificate management on Kubernetes with cert-manager. Let's Encrypt integration, ClusterIssuer configuration, automatic renewal, wildcard https://kubernetes.recipes/recipes/deployments/kubernetes-init-containers-patterns-examples/ https://kubernetes.recipes/recipes/deployments/kubernetes-init-containers-patterns-examples/ Mon, 01 Jun 2026 00:00:00 GMT Deployments Use Kubernetes init containers for pod initialization. Wait for dependencies, clone Git repos, setup configuration, database migrations, certificate https://kubernetes.recipes/recipes/configuration/kubernetes-kind-local-development-cluster/ https://kubernetes.recipes/recipes/configuration/kubernetes-kind-local-development-cluster/ Mon, 01 Jun 2026 00:00:00 GMT Configuration Create local Kubernetes clusters with kind (Kubernetes in Docker). Multi-node clusters, ingress setup, local registry, port mapping, volume mounts, and CI/CD https://kubernetes.recipes/recipes/configuration/kubernetes-kustomize-configuration-management/ https://kubernetes.recipes/recipes/configuration/kubernetes-kustomize-configuration-management/ Mon, 01 Jun 2026 00:00:00 GMT Configuration Manage Kubernetes configurations with Kustomize. Build overlays for multiple environments, patch resources, generate ConfigMaps and Secrets, and integrate