Root cause: → No CloudWatch alarm was configured on DatabaseConnections. The only production alert path was the BetterStack health check, which by then was already failing.{{ROOT_CAUSE}}

3.5.2 Contributing Factors

3.3 What Worked Well

• BetterStack detection was excellent: < 30 seconds from failure to alert.
• Slack alert delivery was immediate: Alert to #drop-ops within 30 seconds.
• App Runner restart is fast and reliable: Recovery completed in < 5 minutes.
• RDS was not the problem: Direct psql connection succeeded, quickly ruling out infrastructure failure.
• No data loss: Audit logs intact, no transactions corrupted.

4. Impact Analysis

5. Corrective Actions

5.1 Immediate (before next marketing campaign)

5.2 Before v1.0 Launch

5.3 ProcessTrigger Event

The specific trigger for this incident: {{TRIGGER}}

6. What Went Well

1. {{CATEGORY_1}}: {{DESCRIPTION}}
2. {{CATEGORY_2}}: {{DESCRIPTION}}
3. {{CATEGORY_3}}: {{DESCRIPTION}}

7. What Went Wrong

1. {{CATEGORY_1}}: {{DESCRIPTION}}
2. {{CATEGORY_2}}: {{DESCRIPTION}}
3. {{CATEGORY_3}}: {{DESCRIPTION}}

8. Where We Got Lucky

1. {{LUCKY_1}}
2. {{LUCKY_2}}
3. {{LUCKY_3}}

9. Action Items

Short-Term Fixes (ongoing)This Sprint)

6. Systemic

Long-Term Improvements

6.1 Connection Pooling Fix

Current state: Implicit pool, no limits, no timeout.

Target state:

// src/drop-app/src/lib/db.ts (example)Next const pool = new Pool({
  connectionString: process.env.DATABASE_URL,
  max: 10,                    // Hard cap — never exceed RDS t4g.micro limit
  idleTimeoutMillis: 30000,   // Release idle connections after 30s
  connectionTimeoutMillis: 2000, // Fail fast if pool is exhausted
});

When PgBouncer is added, set max higher in the app and let PgBouncer enforce the RDS limit.

6.2 CloudWatch AlarmQuarter)

aws cloudwatch put-metric-alarm \
  --alarm-name "drop-db-connections-high" \
  --alarm-description "RDS DatabaseConnections > 70" \
  --metric-name DatabaseConnections \
  --namespace AWS/RDS \
  --dimensions Name=DBInstanceIdentifier,Value=drop-db \
  --period 60 \
  --evaluation-periods 2 \
  --threshold 70 \
  --comparison-operator GreaterThanThreshold \
  --statistic Average \
  --alarm-actions arn:aws:sns:eu-west-1:324480209768:drop-ops-alerts \
  --region eu-west-1

6.3 Marketing Campaign Checklist (Pre-Launch)

Before any marketing campaign that targets > 100 recipients:

•  Notify infrastructure (Alem) at least 24h before send
•  Check current DatabaseConnections baseline in CloudWatch
•  Verify pool configuration is explicit
•  Consider sending campaign in batches (< 100/hour) to spread load

7. Lessons Learned

1. Explicit is always better than implicit for resource limits. Never rely on OS defaults for connection pool configuration in production.
2. Metrics must lead alerts, not lag them. The health check failure was a lagging indicator. DatabaseConnections CloudWatch alarm would have caught this 2 minutes earlier.
3. Marketing and infrastructure must coordinate. A 500-recipient email burst is a load event that needs infrastructure awareness.
4. App Runner restart is a fast, reliable mitigation. < 5 minutes RTO is acceptable for this class of issue. Document it as a first-response step in the runbook.
5. BetterStack + Slack alerting works. < 30 second detection time met our target. No changes needed here.
6. DB-level connection pooling (PgBouncer) is required for burst tolerance at scale. This should be resolved before public launch.

8. Action Item Tracking

Process Changes

10. Follow-Up Tracking

Follow-up review date: {{FOLLOWUP_DATE}} (4 weeks after incident) Follow-up owner: {{FOLLOWUP_OWNER}}

11. Recurrence Prevention

Before this incident: {{BEFORE_STATE}}

After implementing action items: {{AFTER_STATE}}

Confidence in prevention: {{CONFIDENCE}} / 10 Residual risk: {{RESIDUAL_RISK}}

12. Review & Sign-Off

Post-mortem presented at: {{MEETING}} on {{MEETING_DATE}} Meeting recording: {{RECORDING_LINK}} Meeting notes: {{NOTES_LINK}}

Related Documents

• Incident Report INC-2026-001{{ID}}
• Operational Runbook
• Disaster Recovery Plan
• Monitoring & Observability

Approval