ALL
POSTS
59 posts so far.
March 17, 2026CI/CD9 min read
Our Blue-Green Deploy Corrupted 3,200 Records in 11 Minutes
We had blue-green deployments, health checks, and automatic rollbacks. Then a single column rename executed during the deploy window and our old containers spent 11 minutes writing to a column that no longer existed.
March 15, 2026CI/CD9 min read
How a GitHub Actions Cache Hit Skipped Our Tests and Shipped a Regression to 12,000 Users
Our CI pipeline showed green for six consecutive deploys while never running the new test files we added. A cache key tied only to package-lock.json silently restored stale compiled test artifacts — new tests never compiled, never ran, and a broken discount-code checkout reached production for 6 hours.
March 5, 2026CI/CD9 min read
We Deployed on a Friday. Here's What Happened Next.
A production deploy at 4:30 PM on a Friday turned a routine release into a 6-hour incident — and permanently changed how I think about automation and discipline.