Network Connectivity Issues

Overview

“Can’t connect to X” can fail at any of several layers — the local network interface, routing, DNS, a firewall (local or in-path), or the remote service itself being down. Working through them in order avoids chasing the wrong layer.

The Problem

An application can’t reach a database, an external API, or another server, and the exact failure point isn’t immediately obvious.

Investigation

# 1. Local interface
ip a                       # is the interface up, with the expected IP?
ip link show                 # link state

# 2. DNS resolution
getent hosts target-host       # resolve exactly as the application would
dig +short target-host           # bypass /etc/hosts, query DNS directly — compare the two!
cat /etc/resolv.conf                # which DNS servers are configured

# 3. Routing
ip route get <destination_ip>          # which route/interface would be used
traceroute target-host                    # where does it stop responding?

# 4. Port reachability
nc -zv target-host 443                        # is the TCP port actually open?
curl -v https://target-host                      # full connection detail, including TLS handshake

# 5. Local firewall
sudo iptables -L -n -v                              # any DROP/REJECT rules matching?
sudo nft list ruleset

# 6. Remote service health (if you have access)
ssh target-host "systemctl status the-service"

Root Cause Possibilities

• DNS misconfiguration — wrong resolver, or a stale /etc/hosts entry silently overriding real DNS.
• A local or in-path firewall rule — iptables/nftables/security-group rules blocking the specific port.
• Routing misconfiguration — no route to the destination network, or a wrong default gateway.
• The remote service is actually down — not a networking problem at all, just a downstream outage.
• ICMP blocked, but TCP fine (or vice versa) — ping failing is NOT proof of an outage; always test the actual protocol/port the application uses.
• MTU/fragmentation issues — connections establish but larger payloads hang or drop (less common, but a real cause of “works sometimes”).

Resolution

# DNS issue — check for a stale /etc/hosts entry first
grep target-host /etc/hosts
sudo sed -i '/target-host/d' /etc/hosts      # remove a stale entry if found

# Firewall blocking the port
sudo iptables -I INPUT -p tcp --dport 443 -j ACCEPT      # (adjust to your actual security policy — don't blanket-open)

# Routing issue
sudo ip route add <network>/<cidr> via <gateway>

Prevention

• Standardize on curl/nc for health checks in monitoring, not ping alone — ICMP being blocked is common and unrelated to real service health.
• Keep /etc/hosts clean and audited, especially in container base images where a stray entry can silently override DNS for every container built from it.
• Document expected routes/firewall rules as code (Terraform, Ansible) so drift is caught in review rather than discovered during an incident.
• Use mtr instead of one-shot traceroute when diagnosing intermittent connectivity — it shows loss/latency trends over time, not just a single snapshot.

Summary

Network troubleshooting is a layered process: interface → DNS → routing → port reachability → firewall → remote service health. Testing each layer in order, rather than guessing, quickly isolates where the failure actually is — and a failed ping should never be treated as proof of an outage on its own, since ICMP and the application’s real protocol often behave completely differently through firewalls.

Pro Tip

When dig shows the correct IP but the application still can’t connect, check /etc/hosts immediately — it’s checked before DNS for most resolver configurations (/etc/nsswitch.conf), and a stale entry there is invisible to dig/nslookup but fully visible to the actual application.