Processes & Background Jobs

Running a Command in the Background

long_task.sh &            # runs in the background, script continues immediately
echo "Started with PID $!"    # $! is the PID of the last backgrounded process

wait $!                          # block until that specific background job finishes
echo "Background task done"

Running Multiple Tasks in Parallel

#!/usr/bin/env bash
set -euo pipefail

for server in web1 web2 web3; do
    deploy_to.sh "$server" &     # launch each deploy in the background
done

wait                                # wait for ALL background jobs to finish
echo "All deployments complete"

Capturing Exit Status of Parallel Jobs

#!/usr/bin/env bash
PIDS=()
for server in web1 web2 web3; do
    deploy_to.sh "$server" &
    PIDS+=($!)
done

FAILED=0
for pid in "${PIDS[@]}"; do
    if ! wait "$pid"; then
        echo "PID $pid failed" >&2
        FAILED=1
    fi
done

[ "$FAILED" -eq 0 ] && echo "All succeeded" || { echo "Some deployments failed"; exit 1; }

wait alone can’t tell you WHICH job failed — waiting on each PID individually is the correct pattern when you need per-job results.

Limiting Parallelism

#!/usr/bin/env bash
MAX_JOBS=4

for file in *.log; do
    process_log.sh "$file" &
    while (( $(jobs -r | wc -l) >= MAX_JOBS )); do
        wait -n     # wait for at least ONE job to finish before launching more
    done
done
wait

This caps concurrency at MAX_JOBS — important when processing hundreds of files/servers to avoid exhausting CPU, memory, or network connections.

GNU parallel / xargs -P (Better Tools for Heavy Parallelism)

# xargs with parallelism — often cleaner than manual job control for simple cases
find . -name "*.log" | xargs -P 4 -I{} process_log.sh {}

# GNU parallel (if installed) — richer feature set (progress bars, retries, etc.)
parallel -j4 process_log.sh ::: *.log

Capturing Output From a Background Job

long_task.sh > output.log 2>&1 &
PID=$!
wait "$PID"
cat output.log

Background jobs’ stdout/stderr aren’t automatically visible — always redirect them to a file if you need the output later.

Detaching a Process From the Terminal

nohup long_task.sh > output.log 2>&1 &      # survives the terminal/SSH session closing
disown                                          # remove it from this shell's job table entirely

setsid long_task.sh > output.log 2>&1 < /dev/null &   # fully detach into a new session

Checking Running Jobs

jobs                 # background jobs started by THIS shell
jobs -l                 # + PIDs
jobs -r                   # only running jobs
ps aux | grep script.sh     # check from outside the shell that launched it

Production Considerations

• Always cap parallelism (MAX_JOBS, xargs -P N) when fanning out to many servers/files — unbounded parallel execution can exhaust file descriptors, memory, or hit API rate limits.
• Redirect background job output to a file — otherwise it’s lost or interleaves confusingly with the parent script’s own output.
• For anything more complex than a handful of parallel tasks, consider xargs -P/GNU parallel over hand-rolled job control — they handle edge cases (failures, output ordering) more robustly.

Quick Interview Answer

“command & backgrounds a job, $! captures its PID, and wait blocks until it (or all background jobs) complete. For parallel work across many servers/files, launching several backgrounded commands and collecting each PID lets you wait on each individually to know exactly which ones failed — plain wait alone only tells you they’re all done, not which succeeded. xargs -P N or GNU parallel are usually cleaner than hand-rolled job control for anything beyond a few concurrent tasks.”

Common Mistakes

• Using a bare wait after launching parallel jobs and assuming it reports which one failed — it doesn’t, without waiting on each PID individually.
• Launching unlimited parallelism against many targets, overwhelming a shared resource (API rate limits, file descriptors, network bandwidth).
• Not redirecting a backgrounded job’s output, losing error messages needed to debug a failure after the fact.