Scenario Advanced Aws AWS Security

Developer Pushed AWS Credentials to Public GitHub — Incident Response

A developer accidentally committed live AWS access keys to a public GitHub repository. Walk through the immediate containment, investigation, and prevention strategy.

January 20, 2025 4 min read ~30 min to complete DB
The Situation

It's 10:30 AM. Your security monitoring fires: 'GitHub Secret Scanning detected AWS credentials in a public repository.' The keys belong to a CI user with broad S3 and EC2 read/write access. The commit was pushed 47 minutes ago. You don't know if the keys have already been used by someone else.

6 Steps
6 Services Used
~30 min Duration
Advanced Difficulty

The Problem

Leaked AWS credentials are one of the most dangerous security incidents. Attackers have automated scrapers that monitor GitHub in real time — your keys can be picked up within minutes of a push. Every second you don’t act is a second an attacker could be spinning up crypto miners, exfiltrating data, or creating persistence backdoors.

Clock Is Ticking
GitHub research found that 80% of leaked secrets are accessed by automated bots within the first 4 seconds of being pushed publicly. Your window is tiny.

Step 1: Revoke the Keys Immediately (First 2 Minutes)

Do not investigate first. Revoke first.

# Deactivate the key (reversible — do this first)
aws iam update-access-key \
  --access-key-id AKIAIOSFODNN7EXAMPLE \
  --status Inactive \
  --user-name ci-deploy-user

# Then delete it permanently
aws iam delete-access-key \
  --access-key-id AKIAIOSFODNN7EXAMPLE \
  --user-name ci-deploy-user

If the compromised credential was an IAM role (not a user), issue a policy that denies everything immediately:

aws iam put-role-policy \
  --role-name compromised-ci-role \
  --policy-name EmergencyDenyAll \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Deny",
      "Action": "*",
      "Resource": "*"
    }]
  }'

Step 2: Check CloudTrail for Unauthorized Usage

Now investigate what happened during the 47-minute window:

# Look up all API calls made with this access key
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIAIOSFODNN7EXAMPLE \
  --start-time "2024-01-15T09:43:00Z" \
  --end-time "2024-01-15T10:30:00Z" \
  --query 'Events[*].{Time:EventTime,Event:EventName,IP:CloudTrailEvent}' \
  --output table

Look for these red flags in the output:

Suspicious ActionWhat It Means
RunInstancesAttacker spun up EC2 (likely crypto miner)
CreateUser / CreateAccessKeyAttacker created persistence
GetObject on sensitive bucketsData exfiltration
AssumeRoleLateral movement to other roles
Requests from unusual IPs/regionsClear attacker activity
# Pull the raw CloudTrail event to see the source IP
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIAIOSFODNN7EXAMPLE \
  --query 'Events[*].CloudTrailEvent' | python3 -c "
import json, sys
for line in json.load(sys.stdin):
    event = json.loads(line)
    print(event.get('sourceIPAddress'), event.get('eventName'), event.get('awsRegion'))
"

Step 3: Assess Blast Radius

Based on what CloudTrail shows, determine the scope:

# If RunInstances was called — find and terminate attacker's instances
aws ec2 describe-instances \
  --filters "Name=tag:CreatedBy,Values=AKIAIOSFODNN7EXAMPLE" \
  --query 'Reservations[*].Instances[*].InstanceId'

# Check all regions — attackers always use unexpected regions
for region in $(aws ec2 describe-regions --query 'Regions[*].RegionName' --output text); do
  echo "Checking $region..."
  aws ec2 describe-instances --region $region \
    --filters "Name=instance-state-name,Values=running" \
    --query 'Reservations[*].Instances[*].{ID:InstanceId,Region:Placement.AvailabilityZone,Launch:LaunchTime}' \
    --output table
done
# Check for new IAM users or keys created during the window
aws iam list-users --query 'Users[?CreateDate>=`2024-01-15T09:43:00Z`]'
aws iam list-access-keys --query 'AccessKeyMetadata[?CreateDate>=`2024-01-15T09:43:00Z`]'

Step 4: Check GuardDuty Findings

If GuardDuty is enabled, it will have already flagged anomalous activity:

aws guardduty list-findings \
  --detector-id $(aws guardduty list-detectors --query 'DetectorIds[0]' --output text) \
  --finding-criteria '{
    "Criterion": {
      "updatedAt": {
        "GreaterThanOrEqual": 1705312980000
      }
    }
  }'

GuardDuty finding types to look for:

  • UnauthorizedAccess:IAMUser/MaliciousIPCaller
  • UnauthorizedAccess:IAMUser/TorIPCaller
  • Recon:IAMUser/MaliciousIPCaller
  • CryptoCurrency:EC2/BitcoinTool.B

Step 5: Rotate Everything That Was Accessible

Even if you find no evidence of attacker access, rotate defensively:

# Rotate the CI user's credentials (generate new key for pipeline)
aws iam create-access-key --user-name ci-deploy-user

# If CI user had access to RDS — rotate the DB password
aws secretsmanager rotate-secret \
  --secret-id prod/rds/ci-user-password \
  --force-delete-without-recovery

Update your CI/CD pipeline (GitHub Actions secrets, etc.) with the new credentials — and this time, use OIDC federation instead of long-lived keys:

# GitHub Actions — OIDC (no stored credentials at all)
- name: Configure AWS credentials
  uses: aws-actions/configure-aws-credentials@v4
  with:
    role-to-assume: arn:aws:iam::123456789012:role/github-ci-role
    aws-region: us-east-1

Step 6: Prevention Strategy

LayerToolWhat It Does
Developer Machinedetect-secrets pre-commit hookBlocks the commit before it happens
CI PipelinetruffleHog / GitGuardianScans every PR for secrets
RepositoryGitHub Secret ScanningAuto-revokes detected AWS keys
AWS NativeIAM Access AnalyzerFlags overly permissive policies
MonitoringGuardDutyDetects anomalous API activity in real time
ArchitectureOIDC instead of keysEliminates stored credentials entirely

Set up the pre-commit hook:

pip install detect-secrets
detect-secrets scan > .secrets.baseline

# .pre-commit-config.yaml
repos:
- repo: https://github.com/Yelp/detect-secrets
  rev: v1.4.0
  hooks:
  - id: detect-secrets
    args: ['--baseline', '.secrets.baseline']
The Real Fix: Never Use IAM User Keys for CI
The only permanent fix is replacing IAM user long-lived keys with OIDC identity federation. GitHub Actions, GitLab CI, and CircleCI all support it. With OIDC, there are no credentials to leak — the CI system gets a short-lived token valid for a single job.

Post-Incident Report Template

Incident: Leaked AWS Credentials
Date: [date]
Duration: 47 minutes (push to revocation)
Severity: High

Timeline:
  09:43 - Keys pushed to public repo
  10:30 - GitHub Secret Scanning alert triggered
  10:32 - Keys revoked
  10:45 - CloudTrail investigation complete (no attacker usage found)

Root Cause: Developer hardcoded credentials in .env file committed to repo

Corrective Actions:
  1. All CI pipelines migrated to OIDC (due: this sprint)
  2. detect-secrets pre-commit hook added to all repos (due: this week)
  3. GuardDuty real-time alert to PagerDuty configured (complete)
  4. IAM Access Analyzer enabled on all accounts (complete)
Services Used
IAMCloudTrailGuardDutySecrets ManagerGitHub Advanced SecuritySNS
Prerequisites
  • Basic understanding of IAM users and access keys
  • Familiarity with CloudTrail for audit logging
What You Learned
  • The exact sequence of actions to take in the first 15 minutes
  • How to use CloudTrail to determine if keys were used by an attacker
  • How to assess blast radius from unauthorized API calls
  • A layered prevention strategy using pre-commit hooks, CI scanning, and native AWS tools

Have a similar scenario to share?

Production incidents are the best teachers. Submit your real-world scenario and help others learn.

Open Google Form

Related Scenarios

Learning Paths beginner

AWS Cloud Engineer Learning Path

Who Is This Path For? This path is designed for complete beginners who want to break into cloud computing as an AWS engineer. If you know …

Jan 20, 2025 Read more