Guide Bash-Scripting Advanced

Bash + AWS CLI

Scripting AWS operations with the AWS CLI and jq — EC2 automation, S3 sync, tagging audits, and safe patterns for infrastructure scripts.

3 min read

Prerequisites: CLI Configured, jq Installed

aws sts get-caller-identity     # confirm which account/role your credentials belong to
aws configure list                # show current profile/region config

Every serious AWS CLI script pairs with jq, since --output json is the most reliable format to parse programmatically (text/table output is meant for humans and changes column layout more easily).

Listing and Filtering Resources

# Get running EC2 instance IDs with a specific tag
aws ec2 describe-instances \
    --filters "Name=instance-state-name,Values=running" "Name=tag:Team,Values=platform" \
    --query 'Reservations[*].Instances[*].InstanceId' \
    --output text

# Using jq instead of --query for more complex filtering
aws ec2 describe-instances --output json | \
    jq -r '.Reservations[].Instances[] | select(.State.Name=="running") | .InstanceId'

--query (JMESPath) is built into the AWS CLI and works without any extra dependency; jq is often preferred for more complex logic since it’s a full JSON query language.

A Real Script: Stop All Untagged Dev Instances

#!/usr/bin/env bash
set -euo pipefail

REGION="us-east-1"

INSTANCE_IDS=$(aws ec2 describe-instances \
    --region "$REGION" \
    --filters "Name=instance-state-name,Values=running" \
    --query "Reservations[].Instances[?!not_null(Tags[?Key=='Environment'])].InstanceId" \
    --output text)

if [ -z "$INSTANCE_IDS" ]; then
    echo "No untagged running instances found"
    exit 0
fi

echo "Stopping untagged instances: $INSTANCE_IDS"
aws ec2 stop-instances --region "$REGION" --instance-ids $INSTANCE_IDS

S3 Sync and Backup Automation

#!/usr/bin/env bash
set -euo pipefail

BUCKET="s3://my-backups"
DATE=$(date +%Y-%m-%d)

aws s3 sync /data/ "$BUCKET/$DATE/" --delete --exclude "*.tmp"
echo "Backup uploaded to $BUCKET/$DATE/"

# Clean up backups older than 30 days
CUTOFF=$(date -d '30 days ago' +%Y-%m-%d)
aws s3 ls "$BUCKET/" | while read -r line; do
    FOLDER_DATE=$(echo "$line" | awk '{print $2}' | tr -d '/')
    if [[ "$FOLDER_DATE" < "$CUTOFF" ]]; then
        echo "Deleting old backup: $FOLDER_DATE"
        aws s3 rm "$BUCKET/$FOLDER_DATE/" --recursive
    fi
done

Waiting for an AWS Operation to Complete

INSTANCE_ID="i-0123456789abcdef0"
aws ec2 run-instances --image-id ami-xxxx --instance-type t3.micro --query 'Instances[0].InstanceId' --output text

aws ec2 wait instance-running --instance-ids "$INSTANCE_ID"    # AWS CLI's built-in waiters — prefer these over hand-rolled polling
echo "Instance is now running"

The CLI’s built-in wait subcommands (instance-running, instance-stopped, stack-create-complete, etc.) handle polling and backoff correctly — always prefer them over a hand-rolled while loop calling describe-* repeatedly.

Assuming a Role From a Script

CREDS=$(aws sts assume-role --role-arn "arn:aws:iam::123456789012:role/DeployRole" --role-session-name "deploy-$(date +%s)" --output json)

export AWS_ACCESS_KEY_ID=$(echo "$CREDS" | jq -r '.Credentials.AccessKeyId')
export AWS_SECRET_ACCESS_KEY=$(echo "$CREDS" | jq -r '.Credentials.SecretAccessKey')
export AWS_SESSION_TOKEN=$(echo "$CREDS" | jq -r '.Credentials.SessionToken')

aws sts get-caller-identity    # confirm you're now acting as the assumed role

Cost/Resource Audit Script

#!/usr/bin/env bash
set -euo pipefail

echo "=== Unattached EBS Volumes ==="
aws ec2 describe-volumes --filters "Name=status,Values=available" \
    --query 'Volumes[*].[VolumeId,Size,CreateTime]' --output table

echo "=== Unused Elastic IPs ==="
aws ec2 describe-addresses --query 'Addresses[?AssociationId==null].[PublicIp,AllocationId]' --output table

Production Considerations

  • Always specify --region explicitly in scripts (or set AWS_DEFAULT_REGION) rather than relying on an assumed CLI default — scripts run in CI/CD or on a different host may not share the same default configuration.
  • Prefer the AWS CLI’s built-in wait subcommands over hand-rolled polling loops — they implement correct backoff and known state-transition logic.
  • Never hardcode AWS credentials in a script — use IAM roles (EC2 instance profiles, ECS task roles, assume-role) or environment variables injected by the CI/CD system’s secret store.

Quick Interview Answer

“AWS CLI scripts pair aws ec2/s3/... commands with --query (JMESPath) or jq to extract exactly the data needed, since JSON is the most reliably parseable output format. Bulk operations follow a filter-then-act pattern: describe resources matching a filter, extract IDs, then act on them in a loop or a single batch call. The CLI’s built-in wait subcommands handle polling for state changes correctly and should be preferred over hand-rolled while loops calling describe-* repeatedly.”

Common Mistakes

  • Parsing --output text/table output with fragile awk/cut instead of using --query/jq on JSON output.
  • Hand-rolling a polling loop instead of using the CLI’s built-in aws ec2 wait ... commands.
  • Hardcoding AWS access keys directly in a script instead of using IAM roles or a secrets manager.

Add More Questions to This Guide

Know a question that should be here? Share it and help the community!

Open Google Form