Guide Bash-Scripting Intermediate

Text Processing

Using grep, sed, awk, and cut inside Bash scripts to parse logs, transform config files, and build reports.

3 min read
bashscriptingtext-processingawksed

Why This Matters for Bash Scripting

Bash itself has weak native string-manipulation tools for complex tasks — real scripts almost always shell out to grep, sed, awk, and cut, capturing their output with command substitution.

ERROR_COUNT=$(grep -c "ERROR" app.log)
echo "Found $ERROR_COUNT errors"

TOP_IP=$(awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -1 | awk '{print $2}')
echo "Most frequent IP: $TOP_IP"

Extracting Data From Structured Output

# Parse a CSV-style line
LINE="alice,32,engineer"
NAME=$(echo "$LINE" | cut -d, -f1)
AGE=$(echo "$LINE" | cut -d, -f2)

# Parse key=value config lines
while IFS='=' read -r KEY VALUE; do
    echo "Config: $KEY -> $VALUE"
done < <(grep -v '^#' config.env | grep '=')

Building a Log Analyzer

#!/usr/bin/env bash
LOGFILE="access.log"

echo "=== Top 5 IPs ==="
awk '{print $1}' "$LOGFILE" | sort | uniq -c | sort -rn | head -5

echo "=== Status Code Breakdown ==="
awk '{print $9}' "$LOGFILE" | sort | uniq -c | sort -rn

echo "=== 404 Errors ==="
grep " 404 " "$LOGFILE" | awk '{print $7}' | sort | uniq -c | sort -rn

Transforming Config Files In-Place

sed -i "s/^ENVIRONMENT=.*/ENVIRONMENT=production/" .env    # replace a config line
sed -i '/^#/d' config.conf                                   # strip comment lines
sed -i "s/{{VERSION}}/$NEW_VERSION/g" deployment.yaml           # template substitution

Using awk for Reports and Aggregation

# Sum a column
awk '{sum += $3} END {print "Total:", sum}' data.csv

# Filter rows, format output
awk -F, '$2 > 30 {printf "%-10s %s\n", $1, $2}' data.csv

# Multiple conditions
awk -F, '$3 == "engineer" && $2 > 25 {print $1}' employees.csv

Combining Bash Logic With Text Tools

#!/usr/bin/env bash
set -euo pipefail

THRESHOLD=100
ERROR_COUNT=$(grep -c "ERROR" /var/log/app.log)

if [ "$ERROR_COUNT" -gt "$THRESHOLD" ]; then
    echo "ALERT: $ERROR_COUNT errors found (threshold: $THRESHOLD)" >&2
    TOP_ERRORS=$(grep "ERROR" /var/log/app.log | awk -F': ' '{print $2}' | sort | uniq -c | sort -rn | head -5)
    echo "Top error types:"
    echo "$TOP_ERRORS"
    exit 1
fi
echo "Error count OK ($ERROR_COUNT/$THRESHOLD)"

Capturing Multi-Line Output Correctly

RESULT=$(awk '{print $1}' file.txt)      # multi-line output, stored with embedded newlines preserved
echo "$RESULT"                              # MUST quote when printing to preserve line breaks

Production Considerations

  • When embedding sed/awk patterns that include shell variables, be careful with quoting — mixing single and double quotes incorrectly is a common source of scripts that work interactively but fail when run non-interactively.
  • Always quote command substitution results (echo "$RESULT") that may contain multiple lines — unquoted, they get word-split and lose their line breaks.
  • Test destructive sed -i transformations against a copy first, or use sed -i.bak to keep an automatic backup during rollout.

Quick Interview Answer

“Bash scripts lean heavily on grep/sed/awk/cut for real text processing, capturing their output via command substitution ($(...)). awk is best for column-based reports and aggregation, sed for in-place transformation of config files, and grep -c/wc -l for simple counting — combined with Bash’s own if/while logic to turn raw log analysis into automated alerting or reporting scripts.”

Common Mistakes

  • Forgetting to quote a multi-line command substitution result, losing line breaks when printed or passed along.
  • Writing fragile sed/awk one-liners without testing against edge cases (empty fields, unexpected delimiters) before running them in production.
  • Reaching for a full Python/Perl script when a short awk/grep pipeline inside the existing Bash script would be simpler and faster.

Add More Questions to This Guide

Know a question that should be here? Share it and help the community!

Open Google Form