DNS Resolution Failures

Overview

“Can’t resolve hostname” or intermittent DNS failures are common, and the resolution order Linux actually uses (/etc/hosts before DNS, per /etc/nsswitch.conf) is the source of a large fraction of confusing cases.

The Problem

An application fails with a DNS resolution error, or resolves to the wrong IP, even though the domain seems to work fine from elsewhere.

Investigation

# Resolve exactly as the SYSTEM would, including /etc/hosts
getent hosts target-host

# Bypass /etc/hosts and query DNS directly — compare the two results!
dig +short target-host
nslookup target-host

cat /etc/hosts                  # look for a stray/stale entry FIRST
cat /etc/resolv.conf               # which DNS servers are configured
cat /etc/nsswitch.conf                # confirms the actual resolution order (files before dns, typically)

# Test against a specific DNS server directly, bypassing local config entirely
dig @8.8.8.8 target-host

# Check response time — a slow-but-working resolver looks very different from a genuinely broken one
dig target-host | grep "Query time"

Root Cause Possibilities

• A stale /etc/hosts entry — silently overrides real DNS for that hostname; invisible to dig/nslookup, fully visible to the application. Extremely common in container base images with a leftover entry.
• Misconfigured or unreachable DNS servers — /etc/resolv.conf points at a resolver that’s down, unreachable, or simply wrong for the environment.
• DNS server actually returning a stale/wrong record — a recent DNS change hasn’t propagated, or a caching layer (systemd-resolved, a local dnsmasq) is serving an old cached answer past its TTL.
• Search domain misconfiguration — an incomplete hostname resolving against the wrong search domain, producing an unexpected FQDN.
• Container/Kubernetes-specific: a broken CoreDNS/kube-dns pod, or a Pod’s dnsPolicy misconfigured for its actual networking mode.

Resolution

# Stale /etc/hosts entry
sudo sed -i '/target-host/d' /etc/hosts

# Wrong/unreachable DNS server
sudo vi /etc/resolv.conf          # correct the nameserver entries
# On systemd-resolved systems, check the actual live config instead of assuming resolv.conf is authoritative:
resolvectl status

# Clear a local DNS cache (systemd-resolved)
sudo systemd-resolve --flush-caches
sudo systemctl restart systemd-resolved

# Kubernetes: check CoreDNS pods specifically
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns

Prevention

• Keep /etc/hosts minimal and audited, especially in golden/base container images — a stray development-time entry baked into an image affects every container built from it.
• Set explicit, monitored DNS server health checks separate from general application health checks — a slowly degrading resolver causes intermittent, hard-to-reproduce failures.
• Understand your platform’s actual resolution chain (systemd-resolved, dnsmasq, CoreDNS) rather than assuming /etc/resolv.conf is always the final authority — on many modern systems it’s a stub pointing at a local resolver/cache.
• For Kubernetes workloads, monitor CoreDNS pod health explicitly — it’s a single point of failure for cluster-wide service discovery.

Summary

DNS troubleshooting means comparing what getent hosts (the full system resolution path, including /etc/hosts) returns against what dig/nslookup (pure DNS, bypassing local overrides) returns — a mismatch between the two immediately points at /etc/hosts or a resolution-order issue rather than the DNS servers themselves. When they agree and are still wrong, the investigation shifts to the actual DNS server’s configuration, caching layer, or (in Kubernetes) CoreDNS health.

Pro Tip

Always run getent hosts and dig +short side by side when debugging DNS — if they disagree, you’ve already found the answer (/etc/hosts or resolution order), and you’ve saved yourself from incorrectly troubleshooting the DNS servers themselves for something that was never their fault.