Linux Logs

Where Do Linux Logs Live?

Two systems coexist on most modern distros:

• The systemd journal — a structured, binary log store for the kernel and any service managed by systemd. Read with journalctl.
• /var/log/ — traditional plain-text log files, still used by many applications directly (/var/log/nginx/access.log, /var/log/auth.log).

Reading the systemd Journal

journalctl                      # everything, oldest first
journalctl -r                    # reverse (newest first)
journalctl -u nginx               # only one service's logs
journalctl -f                      # follow live, like tail -f
journalctl -p err                  # only priority "error" or worse
journalctl --since "10 min ago" --until "now"
journalctl -k                       # kernel messages only (like dmesg)
journalctl -o json-pretty | head -40   # structured output — great for scripting

Key Files Under /var/log

Practical Log-Reading Commands

tail -f /var/log/nginx/access.log         # follow live
tail -n 200 /var/log/syslog                 # last 200 lines
grep -i "error" /var/log/syslog | tail -50
zgrep "OOM" /var/log/syslog.*.gz             # search inside rotated, gzipped logs
awk '{print $9}' access.log | sort | uniq -c | sort -rn   # count HTTP status codes
journalctl -u myapp --since today | grep -c ERROR

Log Rotation: logrotate

Without rotation, logs grow forever and eventually fill /var/log — a classic outage cause. logrotate (run daily via cron/systemd timer) compresses, renames, and eventually deletes old logs based on rules in /etc/logrotate.d/.

# /etc/logrotate.d/myapp
/var/log/myapp/*.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    create 0640 myapp myapp
    postrotate
        systemctl reload myapp > /dev/null 2>&1 || true
    endscript
}

sudo logrotate -d /etc/logrotate.d/myapp    # dry run, shows what WOULD happen
sudo logrotate -f /etc/logrotate.d/myapp    # force rotation now

The postrotate/endscript block matters: many applications keep writing to the old, now-renamed file handle until told to reopen it — that’s why nginx/rsyslog need a reload signal after rotation, not just the rename.

Production Considerations

• Set a journal size cap (SystemMaxUse= in /etc/systemd/journald.conf) — an unbounded journal can silently consume significant disk space.
• Ship logs off-box (to CloudWatch, ELK, Loki, Datadog) for anything you need to retain past the local rotation window or correlate across many hosts.
• A process holding a deleted (post-rotation, pre-reload) log file open keeps writing into space that df shows as used but du can’t find — lsof | grep deleted reveals it.

Quick Interview Answer

“Modern Linux logs live in two places: the structured systemd journal, read with journalctl, and traditional plain-text files under /var/log. journalctl -u <service> -f is the everyday tool for live service logs; logrotate prevents log files from growing forever by rotating, compressing, and eventually deleting old ones, and typically needs a postrotate hook to tell the still-running application to reopen its log file.”

Common Mistakes

• Deleting a growing log file directly instead of truncating it (> file.log) or configuring logrotate — the process keeps its file handle, so disk space isn’t actually freed until the process restarts.
• Forgetting the postrotate reload hook, leaving the application writing into a renamed/rotated-away file.
• Relying only on local logs with no shipping/retention strategy, losing evidence needed for a postmortem after a host is terminated (especially relevant for autoscaled or ephemeral infrastructure).