Debug DNS Resolution Failures in Pods

💡 Quick Answer: Test with kubectl exec <pod> -- nslookup kubernetes.default. If it fails, check CoreDNS pods (kubectl get pods -n kube-system -l k8s-app=kube-dns), verify the Service ClusterIP (kubectl get svc kube-dns -n kube-system), and check if NetworkPolicies block UDP/TCP port 53.

The Problem

Pods can’t resolve DNS names — Service names, external domains, or both. Applications fail with “Name or service not known”, “Temporary failure in name resolution”, or connection timeouts when using hostnames.

The Solution

Step 1: Test DNS from Inside a Pod

# Quick test — resolve the kubernetes API service
kubectl exec -it deploy/myapp -- nslookup kubernetes.default
# If this fails → cluster DNS is broken

# Test external resolution
kubectl exec -it deploy/myapp -- nslookup google.com
# If cluster DNS works but external fails → upstream DNS issue

# Use a debug pod if your containers don't have nslookup
kubectl run dns-test --image=busybox:1.36 --rm -it -- nslookup kubernetes.default

Step 2: Check CoreDNS

# Are CoreDNS pods running?
kubectl get pods -n kube-system -l k8s-app=kube-dns
# NAME                       READY   STATUS    RESTARTS
# coredns-5d78c9869d-abc12   1/1     Running   0
# coredns-5d78c9869d-def34   1/1     Running   0

# Check CoreDNS logs for errors
kubectl logs -n kube-system -l k8s-app=kube-dns --since=5m

# Check the DNS Service ClusterIP
kubectl get svc kube-dns -n kube-system
# NAME       TYPE        CLUSTER-IP   PORT(S)
# kube-dns   ClusterIP   10.96.0.10   53/UDP,53/TCP

Step 3: Check Pod DNS Configuration

# See what DNS config the pod has
kubectl exec myapp-pod -- cat /etc/resolv.conf
# nameserver 10.96.0.10         ← Should point to kube-dns ClusterIP
# search myapp.svc.cluster.local svc.cluster.local cluster.local
# options ndots:5               ← Important!

Step 4: The ndots Problem

With ndots:5, any name with fewer than 5 dots is treated as a relative name. google.com (1 dot) triggers 4 search domain lookups BEFORE the absolute query:

1. google.com.myapp.svc.cluster.local → NXDOMAIN
2. google.com.svc.cluster.local → NXDOMAIN
3. google.com.cluster.local → NXDOMAIN
4. google.com. → SUCCESS

This adds latency. Fix with a trailing dot or lower ndots:

spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"    # Reduce unnecessary search domain lookups

Step 5: Check NetworkPolicy

# If you have NetworkPolicies, DNS (port 53) must be allowed
kubectl get networkpolicy -n myapp

# Ensure egress allows DNS
# See: network-policy-debug-connectivity recipe for the allow-dns policy

Common Issues

CoreDNS CrashLooping

kubectl logs -n kube-system -l k8s-app=kube-dns --previous
# Common: "Loop detected" — CoreDNS is forwarding to itself
# Fix: check /etc/resolv.conf on the NODE (not pod) — ensure it doesn't point to the ClusterIP

DNS Works for Services but Not External Names

CoreDNS upstream forwarder may be misconfigured:

kubectl get configmap coredns -n kube-system -o yaml
# Check the "forward" directive — should point to valid upstream DNS
# forward . /etc/resolv.conf   ← Uses node's DNS
# forward . 8.8.8.8 8.8.4.4   ← Explicit upstream

Scaling CoreDNS for High DNS Traffic

If CoreDNS itself is the bottleneck (high latency under load, not misconfiguration), scale it or add a cache layer:

kubectl scale deployment/coredns -n kube-system --replicas=3

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: {name: coredns, namespace: kube-system}
spec:
  scaleTargetRef: {apiVersion: apps/v1, kind: Deployment, name: coredns}
  minReplicas: 2
  maxReplicas: 10
  metrics: [{type: Resource, resource: {name: cpu, target: {type: Utilization, averageUtilization: 70}}}]

For very high-traffic clusters, NodeLocal DNSCache runs a DNS cache on every node, cutting most lookups down to a local hop instead of a trip to a CoreDNS pod:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml

Continuous DNS Health Check

Useful for catching intermittent failures that a one-off nslookup test misses:

apiVersion: v1
kind: Pod
metadata: {name: dns-monitor}
spec:
  containers:
    - name: dns-test
      image: busybox:1.36
      command:
        - /bin/sh
        - -c
        - |
          while true; do
            nslookup kubernetes.default > /dev/null 2>&1 && echo "Internal: OK" || echo "Internal: FAIL"
            nslookup google.com > /dev/null 2>&1 && echo "External: OK" || echo "External: FAIL"
            sleep 10
          done

Best Practices

• Lower ndots to 2 for pods that resolve many external names — reduces DNS queries by 3x
• Use FQDN with trailing dot in configs — api.example.com. skips search domains entirely
• Always allow DNS in NetworkPolicies — UDP+TCP port 53 to kube-dns
• Monitor CoreDNS — dashboard or coredns_dns_request_count_total metric
• Scale CoreDNS or add NodeLocal DNSCache once request volume, not misconfiguration, is the bottleneck
• Don’t use dnsPolicy: Default unless you want node DNS instead of cluster DNS

Key Takeaways

• nslookup kubernetes.default is the first test — if it fails, CoreDNS is down or unreachable
• Check /etc/resolv.conf in the pod — nameserver should be the kube-dns ClusterIP
• ndots:5 causes 4 extra lookups per external name — lower it for latency-sensitive apps
• NetworkPolicies blocking port 53 is a common hidden cause
• CoreDNS logs reveal upstream failures, loop detection, and SERVFAIL causes