Log Processing

Tailing and Filtering Logs Live

tail -f /var/log/app.log | grep --line-buffered "ERROR"    # --line-buffered keeps grep piping live, not batched
tail -f /var/log/app.log | grep -E "ERROR|CRITICAL"

Counting and Summarizing Errors

#!/usr/bin/env bash
LOGFILE="/var/log/app.log"

TOTAL_LINES=$(wc -l < "$LOGFILE")
ERROR_COUNT=$(grep -c "ERROR" "$LOGFILE")
WARN_COUNT=$(grep -c "WARN" "$LOGFILE")

echo "Total lines: $TOTAL_LINES"
echo "Errors: $ERROR_COUNT"
echo "Warnings: $WARN_COUNT"

echo "=== Top 5 error messages ==="
grep "ERROR" "$LOGFILE" | awk -F'ERROR: ' '{print $2}' | sort | uniq -c | sort -rn | head -5

Time-Windowed Log Analysis

# Extract only the last hour's worth of entries (assuming a standard timestamp format)
LAST_HOUR=$(date -d '1 hour ago' '+%Y-%m-%d %H')
grep "^$LAST_HOUR" /var/log/app.log | grep -c "ERROR"

# journalctl handles this natively for systemd-managed services — usually preferable
journalctl -u myapp --since "1 hour ago" | grep -c "ERROR"

Building a Threshold-Based Alert Script

#!/usr/bin/env bash
set -euo pipefail

LOGFILE="/var/log/app.log"
THRESHOLD=50
WEBHOOK_URL="https://hooks.slack.com/services/XXX"

ERROR_COUNT=$(grep "ERROR" "$LOGFILE" | grep -c "$(date '+%Y-%m-%d')")

if [ "$ERROR_COUNT" -gt "$THRESHOLD" ]; then
    MESSAGE="ALERT: $ERROR_COUNT errors today (threshold: $THRESHOLD) in $LOGFILE"
    echo "$MESSAGE" >&2
    curl -sf -X POST -H 'Content-Type: application/json' \
        -d "{\"text\":\"$MESSAGE\"}" "$WEBHOOK_URL"
    exit 1
fi
echo "Error count OK: $ERROR_COUNT/$THRESHOLD"

Extracting Structured Fields From Log Lines

# A typical Apache/nginx access log line:
# 192.168.1.1 - - [20/Jan/2026:10:15:32 +0000] "GET /api/users HTTP/1.1" 200 1234

while IFS= read -r line; do
    IP=$(echo "$line" | awk '{print $1}')
    STATUS=$(echo "$line" | awk '{print $9}')
    PATH_REQUESTED=$(echo "$line" | awk '{print $7}')

    if [ "$STATUS" -ge 500 ]; then
        echo "5xx error: $IP requested $PATH_REQUESTED"
    fi
done < access.log

Log Rotation From a Script (When Not Using logrotate)

LOGFILE="/var/log/app.log"
MAX_SIZE=$((100 * 1024 * 1024))    # 100MB in bytes

CURRENT_SIZE=$(stat -c%s "$LOGFILE")
if [ "$CURRENT_SIZE" -gt "$MAX_SIZE" ]; then
    mv "$LOGFILE" "${LOGFILE}.$(date +%Y%m%d-%H%M%S)"
    touch "$LOGFILE"
    gzip "${LOGFILE}."*[0-9]
    # signal the app to reopen its log file handle, since it's still writing to the OLD inode
    systemctl reload myapp
fi

Note the same lesson from the Linux Logs fundamentals article: after moving/renaming a log file, the application must be told to reopen it (reload/SIGHUP), or it keeps writing into the renamed, no-longer-visible file.

Searching Across Rotated, Compressed Logs

zgrep "ERROR" /var/log/app.log.*.gz    # search inside gzipped rotated logs without full extraction
zcat /var/log/app.log.*.gz | grep "ERROR" | wc -l

Production Considerations

Prefer logrotate (with a proper postrotate reload hook) over hand-rolled rotation logic for anything running long-term — it’s battle-tested and handles edge cases hand-rolled scripts often miss.
Ship parsed alerts (Slack/PagerDuty webhooks) rather than only writing to a local file that nobody actively watches — a threshold-based alert script is only useful if someone actually sees the alert.
Use journalctl --since/-u for systemd-managed services instead of hand-parsing timestamps from plain-text logs — it’s more reliable and handles time-zone/format issues correctly.

Quick Interview Answer

“Log-processing scripts typically combine grep/awk for extraction and counting with a threshold check and an alerting step (webhook, email) — the pattern is: parse, count, compare against a threshold, alert if exceeded. For systemd services, journalctl --since/-u is more reliable than parsing plain-text timestamps by hand. Rotated logs need zgrep/zcat to search without full decompression, and any custom rotation logic must remember to reload the writing application afterward.”

Common Mistakes

Writing custom log rotation logic without reloading the application afterward, so it keeps writing into the old, renamed file.
Hand-parsing timestamps from plain-text logs when journalctl --since would be more robust for systemd-managed services.
Building an alerting script that only logs locally instead of actually notifying someone (Slack, PagerDuty, email).