ALL
POSTS
41 posts so far.
One kafka-consumer-groups.sh Command Sent $180k in Duplicate Payments
A single Kafka offset reset to --to-earliest replayed 3 hours of payment events on a live consumer group. Here is exactly what happened and how we fixed it.
How AWS SQS Visibility Timeout Caused the Same Order to Be Processed 847 Times
A production war story about how a 30-second SQS visibility timeout turned a slow order processor into a duplicate-charge machine — and how we fixed it with heartbeats and a distributed lock.
How a Race Condition in Our Cron Job Sent 2.3 Million Duplicate Emails in One Night
A nightly email digest cron job was running on two servers simultaneously without a distributed lock — what started as a minor scheduling overlap turned into a 2.3 million email catastrophe that got our domain blacklisted before sunrise.
How Next.js 15's Full Route Cache Served Stale Prices at Checkout for 3 Hours
After migrating a SaaS checkout flow to Next.js 15 App Router, our price display layer silently served cached values — not the live database prices — costing us 3 hours of confused customers and 19 manual refunds.
How a Redis Cache Key Missing One Field Leaked Client Data Across Tenants for 72 Hours
A SaaS platform cached API responses by resource ID alone — when two tenants happened to share the same integer ID, one client spent three days reading another's confidential records.
We Killed the PHP Monolith. It Took 18 Months and One Client's Data.
What looked like a clean strangler-fig migration turned into 18 months of session bridges, soft-delete mismatches, and hard lessons about the implicit contracts hiding inside every legacy codebase.