Single AZ Failure Took Down Black Friday — Root Cause & Fix
The Problem Your application went down because critical resources were effectively pinned to a single Availability Zone. Even though AWS …
Jan 20, 2025
Read more
Incident Response
6 articles
Production RDS Ran Out of Storage at 3 AM — Incident Response and RCA
The Problem RDS storage full is a P0 incident. Unlike CPU spikes that throttle gracefully, a full disk makes the database refuse all writes …
Jan 20, 2025
Read more
EC2 Instance Communicating with Malicious IP — Incident Response
The Problem A running EC2 instance may be compromised — malware installed, data being exfiltrated, or the instance being used as a pivot to …
Jan 20, 2025
Read more
Developer Pushed AWS Credentials to Public GitHub — Incident Response
The Problem Leaked AWS credentials are one of the most dangerous security incidents. Attackers have automated scrapers that monitor GitHub …
Jan 20, 2025
Read more
Production RDS Ran Out of Storage at 3 AM — Incident Response and RCA
The Problem RDS storage full is a P0 incident. Unlike CPU spikes that throttle gracefully, a full disk makes the database refuse all writes …
Jan 20, 2025
Read more
EC2 Production Instance Unreachable — Diagnose and Recover
Overview An EC2 instance going unreachable is one of the most common production incidents. The tricky part isn’t fixing it — …
Mar 10, 2024
Read more