Disk Full — No Space Left on Device

Overview

“No space left on device” can mean the disk is genuinely full, the inode table is exhausted (rarer, but df -h won’t show it), or space is used by a file that’s been deleted but is still held open by a running process.

The Problem

Writes are failing, services can’t log or write temp files, and df -h shows a filesystem at or near 100%.

Investigation

df -h                          # which filesystem is actually full?
df -i                             # inode usage — can be exhausted even with free block space

du -sh /* 2>/dev/null | sort -rh | head -10       # largest top-level directories
du -sh /var/log/* | sort -rh | head -10               # logs are the most common culprit

# The classic mismatch check: df shows full, but du doesn't add up
lsof +L1                              # files deleted but still held open by a process — space not actually freed

# Find the biggest files system-wide (careful — can be slow on large filesystems)
find / -xdev -type f -size +500M -exec ls -lh {} \; 2>/dev/null

Root Cause Possibilities

• Unbounded or unrotated logs — an application writing verbosely with no logrotate configuration, or a postrotate reload hook missing so the app keeps writing into a renamed file.
• A deleted file still open — something (often a crashed process restart, or a log rotation without reload) is holding a large deleted file’s inode, so du and df disagree.
• Inode exhaustion — millions of tiny files (a poorly-managed cache or session-file directory) exhaust available inodes long before block space runs out.
• A runaway process writing temp files — a bug generating unbounded temp/cache files in /tmp or an application-specific directory.
• Old kernels/packages never cleaned up — /boot filling from accumulated old kernel versions is a classic, specific case on long-lived servers.

Resolution

# 1. Free immediate space — start with logs, they're almost always safe to truncate/rotate
sudo truncate -s 0 /var/log/hugefile.log      # truncate WITHOUT breaking the process's open file handle
sudo journalctl --vacuum-size=500M               # shrink the systemd journal

# 2. If a deleted-but-open file is the cause, restart the offending process
#    to release the file handle (identified via lsof +L1)
sudo systemctl restart <service>

# 3. Remove genuinely unneeded large files
sudo rm -f /path/to/old_backup.tar.gz

# 4. If /boot specifically is full (inode/space exhaustion from old kernels):
sudo apt autoremove --purge          # Debian/Ubuntu: removes old kernel packages
sudo dnf remove $(dnf repoquery --installonly --latest-limit=-2 -q)   # RHEL/Fedora equivalent, keeps last 2

Never delete a growing log file with rm while the writing process is still running — use truncate -s 0 instead, which empties it while keeping the process’s file handle valid. rm just unlinks the name; the process keeps writing into the (now invisible) deleted file, and the space isn’t actually freed.

Prevention

• Configure logrotate with sane size/age limits and a correct postrotate reload hook for every application that logs to a file (see the Linux Logs fundamentals article).
• Alert on disk usage well before 90%, and separately alert on inode usage (df -i) for directories known to accumulate many small files.
• Periodically audit and clean old kernel packages, unused Docker images/volumes, and stale backups as part of routine maintenance, not only when disk is already full.
• Set journald’s SystemMaxUse= to cap the systemd journal’s disk footprint.

Summary

A full disk is diagnosed with df -h (blocks) and df -i (inodes), with du -sh narrowing down which directory is responsible. When df and du disagree, lsof +L1 almost always reveals a deleted-but-open file as the real cause. The safe immediate fix for a growing log file is truncate, not rm — deleting the name doesn’t free space until the writing process closes or restarts.

Pro Tip

If df -h shows a filesystem 100% full but du -sh / doesn’t come close to matching, stop looking for “missing” files — go straight to lsof +L1. This single command resolves the vast majority of “disk full but I can’t find what’s using it” incidents.