High Memory Usage & OOM Kills

Overview

A process was unexpectedly killed, or the server is grinding to a halt under memory pressure. Linux’s Out-Of-Memory (OOM) killer is a kernel subsystem that terminates processes when the system runs critically low on memory — it’s a last resort, not a bug in the killed application itself.

The Problem

A service crashed with no application-level error, or the whole server became unresponsive before recovering. Logs show nothing obviously wrong at the application layer.

Investigation

free -h                    # check "available", not raw "free" — see Fundamentals: Linux Storage
cat /proc/meminfo             # detailed breakdown straight from the kernel

# Confirm whether the OOM killer actually fired
dmesg -T | grep -i "killed process"
journalctl -k | grep -i "out of memory"
grep -i "oom" /var/log/syslog    # or /var/log/messages, depending on distro

A confirmed OOM kill in dmesg looks like:

Out of memory: Killed process 1234 (java) total-vm:4194304kB, anon-rss:2097152kB

Once confirmed, find what was actually consuming memory at the time (if the process is still running or another one is the real hog):

ps aux --sort=-%mem | head -10       # top memory consumers, current snapshot
smem -tk                                # per-process, accounting for shared memory correctly (if installed)
cat /proc/<PID>/status | grep -i vm      # detailed per-process memory breakdown

Root Cause Possibilities

• A memory leak — a process’s usage grows unbounded over time (see the Memory Leak scenario in Python Scenarios for a concrete example).
• Undersized instance — legitimate workload genuinely needs more RAM than provisioned.
• No memory limits set — a single container/process consumed all available memory, starving everything else on a shared host.
• Swappiness misconfiguration — the kernel started swapping aggressively (vm.swappiness too high) well before memory was truly exhausted, degrading performance before the actual OOM point.
• A sudden traffic spike — legitimate load exceeded capacity all at once (a cache stampede, a batch job launched at the wrong time).

Resolution

# Immediate: identify and restart/scale the affected service
systemctl status <service>          # was it restarted automatically after the OOM kill?
systemctl restart <service>            # if not

# If it's a genuine leak, capture evidence before restarting if possible:
cat /proc/<PID>/smaps_rollup           # memory map summary, useful for leak analysis

# Set explicit memory limits so ONE process can't take down the whole host
# systemd:
systemctl edit myapp.service
# [Service]
# MemoryMax=2G

# Kubernetes: always set resources.limits.memory in the pod spec

Prevention

• Always set explicit memory limits (cgroups, systemd MemoryMax=, Kubernetes resources.limits.memory) — an unconstrained process can consume all host memory and take down unrelated workloads too.
• Monitor “available” memory (not raw “free”) with alerting well before it hits zero, giving time to react before an OOM kill happens.
• For applications with known leak-prone patterns (unbounded caches, module-level collections), add periodic restarts or proper TTL-based eviction as a stopgap while the root cause is fixed.
• Review vm.swappiness for latency-sensitive workloads (databases especially) — high swappiness can mask memory pressure until it’s severe, then it manifests as a sudden performance cliff.

Summary

An OOM kill is a kernel-level event, confirmed via dmesg/journalctl -k, not something visible in application logs alone. Diagnosis means checking free -h’s “available” column, confirming the kill in kernel logs, and identifying what actually consumed the memory — a leak, an undersized host, or a lack of per-process limits. The durable fix is almost always setting explicit memory limits so the failure mode becomes a contained restart, not a host-wide outage.

Pro Tip

kubectl describe pod reporting OOMKilled is Kubernetes surfacing exactly this kernel event — always cross-check the underlying node’s dmesg/kernel logs too, since the pod-level view alone won’t show whether OTHER workloads on the same node were also affected by the same memory pressure event.