Text Processing Commands

The Unix Pipeline Philosophy

Linux text tools are designed to be small and composable — chained together with | (pipe), where each tool’s stdout becomes the next tool’s stdin.

flowchart LR A[cat access.log] --> B["grep 404"] B --> C["awk '{print $1}'"] C --> D["sort"] D --> E["uniq -c"] E --> F["sort -rn"] F --> G["Top offending IPs"]

cat access.log | grep " 404 " | awk '{print $1}' | sort | uniq -c | sort -rn | head

grep: Search Text by Pattern

grep "error" app.log                  # lines containing "error"
grep -i "error" app.log                 # case-insensitive
grep -v "debug" app.log                   # invert match — lines NOT containing "debug"
grep -r "TODO" src/                         # recursive across a directory
grep -n "error" app.log                       # show line numbers
grep -c "error" app.log                         # count matching lines
grep -E "error|warning" app.log                   # extended regex, multiple patterns
grep -A3 -B1 "Exception" app.log                     # 3 lines after, 1 line before each match

sed: Stream Editor (Find & Replace)

sed 's/foo/bar/' file.txt              # replace first occurrence per line
sed 's/foo/bar/g' file.txt               # replace ALL occurrences per line
sed -i 's/foo/bar/g' file.txt              # edit the file IN PLACE (no output, direct modify)
sed -i.bak 's/foo/bar/g' file.txt            # in-place, but keep a .bak backup first
sed -n '5,10p' file.txt                        # print only lines 5–10
sed '/^#/d' config.conf                          # delete comment lines

awk: Column-Based Processing & Reports

awk '{print $1}' access.log                       # print first column (fields split by whitespace)
awk -F: '{print $1}' /etc/passwd                     # custom field separator (colon)
awk '{sum += $NF} END {print sum}' data.txt            # sum the last column
awk '$3 > 100 {print $1, $3}' data.txt                   # filter rows where column 3 > 100
awk '{print NR, $0}' file.txt                              # prepend line numbers

cut, sort, uniq — Small but Essential

cut -d: -f1 /etc/passwd            # extract field 1, using ':' as delimiter
cut -c1-10 file.txt                  # extract characters 1 through 10

sort file.txt                          # alphabetical sort
sort -n numbers.txt                      # numeric sort
sort -r file.txt                           # reverse sort
sort -k2 -t, data.csv                        # sort by 2nd comma-delimited field

uniq file.txt                                  # remove adjacent duplicate lines (needs sorted input!)
uniq -c file.txt                                 # count occurrences of each line
sort file.txt | uniq -c | sort -rn                 # classic "count and rank" combo

tr and xargs

echo "Hello World" | tr 'a-z' 'A-Z'    # translate: uppercase everything
echo "a,b,c" | tr ',' '\n'               # replace commas with newlines

echo "file1.txt file2.txt" | xargs rm      # feed arguments from stdin into a command
find . -name "*.log" | xargs grep "ERROR"    # search across many files found by find
find . -name "*.tmp" -print0 | xargs -0 rm     # -print0/-0 safely handles filenames with spaces

Production Considerations

sed -i without .bak has no undo — always test the pattern without -i first, or keep a backup suffix.
uniq only removes adjacent duplicates — always sort first unless the input is already grouped.
Prefer xargs -0/find -print0 when filenames might contain spaces or newlines — the default whitespace-split behavior silently breaks on them.

Quick Interview Answer

“grep finds lines matching a pattern, sed does find-and-replace (or line-based editing) on a text stream, and awk processes text column-by-column, which makes it ideal for reports and aggregation. They’re built to be piped together — grep | awk | sort | uniq -c is a classic combo for turning a raw log file into a ranked summary.”

Common Mistakes

Running uniq on unsorted input and being confused why duplicates aren’t removed.
Using sed -i in production without a backup or a dry run.
Reaching for a Python script when a one-line awk/grep pipeline would do the job faster and with fewer moving parts.