Slow Disk I/O

Overview

Applications feel slow, but CPU and memory both look fine. The bottleneck is often the disk — either the physical/virtual storage device itself, or a filesystem-level issue like fragmentation or a full disk affecting write performance.

The Problem

A database or application has elevated latency, but top shows low CPU usage and free -h shows healthy memory — the classic signature of an I/O-bound, not compute-bound, problem.

Investigation

iostat -x 1 5              # extended stats, 5 one-second samples

Focus on these columns:

vmstat 1 5                    # check the 'wa' column — % CPU time waiting on I/O
ps aux | awk '$8 ~ /D/'          # processes stuck in uninterruptible sleep (D state) — a strong I/O-wait signal

iotop -o                            # per-process I/O usage, live (if installed) — pinpoints WHICH process is driving I/O

Root Cause Possibilities

• The disk/volume is genuinely saturated — legitimate I/O load has outgrown provisioned IOPS/throughput (especially relevant on cloud EBS/managed disks with hard IOPS caps).
• A single noisy-neighbor process — one process (a backup job, a badly-written query, log spam) is monopolizing I/O bandwidth that other processes need.
• Swapping due to memory pressure — see High Memory Usage: swap activity itself generates disk I/O, compounding the slowdown.
• Filesystem issues — a nearly-full filesystem can degrade write performance well before 100% (allocation becomes harder to do efficiently), or heavy fragmentation on some filesystem types.
• Network storage latency — for NFS or network-attached volumes, the “disk” slowness is really network latency to the storage backend.

Resolution

# Identify and address the specific I/O-heavy process
iotop -o                       # find it
renice -n 19 -p <PID>            # if it's a background/non-critical job, lower its priority
ionice -c 3 -p <PID>               # or set it to "idle" I/O scheduling class — yields to other I/O

# If it's a cloud volume hitting its IOPS/throughput ceiling
# -> provision higher IOPS/throughput, or move to a faster storage tier

# Confirm whether it's really disk, or actually network storage
mount | grep nfs                  # is the slow path on an NFS mount?
findmnt -T /var/lib/mydb            # check the filesystem type backing a specific path

Prevention

• Set IOPS/throughput alerts on cloud-provisioned storage well before hitting the provisioned ceiling, not after applications start timing out.
• Use ionice for known-heavy background jobs (backups, log compaction) so they yield priority to latency-sensitive foreground workloads.
• Keep filesystems below ~85-90% utilization — write performance degrades as free space becomes fragmented and harder to allocate efficiently.
• For databases specifically, separate data and log/WAL volumes onto different disks where possible, so one workload’s I/O pattern doesn’t starve the other.

Summary

Disk I/O bottlenecks show up as high await/%util in iostat -x and processes stuck in D state, with low CPU usage ruling out a compute-bound explanation. iotop narrows it down to a specific process, and the fix is either re-prioritizing that process (ionice/renice), addressing a genuinely undersized storage tier, or discovering the “disk” issue is actually network latency to an NFS/network-attached volume.

Pro Tip

A process in uninterruptible sleep (D state in ps) cannot be killed with kill -9 — the kernel won’t deliver any signal to a process blocked on a low-level I/O operation. If a D-state process seems permanently stuck, the fix is almost always at the storage layer (fixing the underlying slow/unresponsive device or NFS server), not at the process level.