# Disaster Recovery Plan

# Disaster Recovery Plan

> **Project:** {{PROJECT_NAME}}
> **Version:** {{VERSION}}
> **Date:** {{DATE}}
> **Author:** {{AUTHOR}}
> **Status:** Draft | In Review | Approved
> **Reviewers:** {{REVIEWERS}}

## Document History
| Version | Date | Author | Changes |
|---------|------|--------|---------|
| 0.1     | {{DATE}} | {{AUTHOR}} | Initial draft |

---

## 1. Business Continuity Overview

<!-- GUIDANCE: Explain the purpose of this plan and when it applies. Include scope and who is responsible. -->

This plan documents the procedures to recover {{PROJECT_NAME}} services following a disaster event (data center failure, data corruption, security breach, or catastrophic failure).

**Plan Owner:** {{DR_OWNER}}
**Plan Reviewer:** {{DR_REVIEWER}}
**Last Tested:** {{LAST_TEST_DATE}}
**Next Scheduled Test:** {{NEXT_TEST_DATE}}

**Disaster types covered:**
- Infrastructure failure (AZ/region outage)
- Data corruption or accidental deletion
- Security incident (ransomware, data breach)
- Vendor/provider outage
- Catastrophic application failure

---

## 2. RPO / RTO Targets Per Service Tier

<!-- GUIDANCE: RPO = how much data can we lose (time since last backup). RTO = how quickly must we restore service. -->

| Tier | Description | RPO | RTO | Examples |
|------|-------------|-----|-----|----------|
| Tier 1 — Critical | Core user-facing services; downtime has direct revenue impact | 0 (real-time replication) | < 15 min | Auth, checkout, core API |
| Tier 2 — Important | Supporting services; degraded experience without them | < 1 hour | < 4 hours | Notifications, reports |
| Tier 3 — Standard | Background/admin services; business can operate without temporarily | < 24 hours | < 24 hours | Analytics, admin panel |

---

## 3. Service Tier Classification

<!-- GUIDANCE: Assign every service to a tier. Re-review this table when new services are added. -->

| Service | Tier | Owner | Rationale |
|---------|------|-------|-----------|
| {{SERVICE_1}} | Tier 1 | {{OWNER}} | Core user journey |
| {{SERVICE_2}} | Tier 1 | {{OWNER}} | Authentication |
| {{SERVICE_3}} | Tier 2 | {{OWNER}} | Supporting |
| {{SERVICE_4}} | Tier 3 | {{OWNER}} | Admin only |
| Database — Primary | Tier 1 | Platform | All services depend on it |
| Object Storage | Tier 2 | Platform | User uploads |

<!-- TODO: Complete service list -->

---

## 4. Backup Strategy

### 4.1 Database Backups

<!-- GUIDANCE: Document every database, backup method, and how you verify recoverability. -->

| Database | Backup Type | Frequency | Retention | Location | Verified |
|----------|-------------|-----------|-----------|----------|----------|
| {{DB_PRIMARY}} | Automated snapshot | Daily | 30 days | {{BACKUP_LOCATION}} | Monthly |
| {{DB_PRIMARY}} | Point-in-time recovery | Continuous | 7 days | {{BACKUP_LOCATION}} | Monthly |
| {{DB_READ_REPLICA}} | Not backed up separately | — | — | Rebuilt from primary | — |

**Automated backup tool:** {{BACKUP_TOOL}} <!-- AWS RDS automated backups / pg_dump + S3 / Barman -->
**Backup encryption:** AES-256, key managed in {{KMS_TOOL}}
**Cross-region copy:** {{CROSS_REGION}} <!-- Yes — copied to {{DR_REGION}} / No -->

### 4.2 File / Object Storage Backups

| Storage | Backup Method | Frequency | Retention | DR Copy |
|---------|--------------|-----------|-----------|---------|
| {{S3_BUCKET}} | S3 versioning + replication | Continuous | {{RETENTION}} | {{DR_BUCKET}} |
| {{FILE_STORE}} | Snapshot | Daily | 30 days | Cross-region |

### 4.3 Configuration Backups

| Config | Backup Method | Location | Frequency |
|--------|--------------|----------|-----------|
| IaC (Terraform) | Git repository | {{GIT_REPO}} | On change |
| Application config | Git repository | {{GIT_REPO}} | On change |
| Secrets | Secrets manager replication | {{SECRETS_BACKUP}} | Real-time |
| DNS records | Export to Git | {{GIT_REPO}} | Weekly |
| TLS certificates | Secrets manager | {{CERTS_BACKUP}} | On renewal |

### 4.4 Backup Testing Schedule

<!-- GUIDANCE: Untested backups are not backups. Every backup type must be tested periodically. -->

| Backup Type | Test Frequency | Last Test | Result | Tester |
|-------------|---------------|-----------|--------|--------|
| Database full restore | Monthly | {{DATE}} | {{RESULT}} | {{TESTER}} |
| Point-in-time restore | Quarterly | {{DATE}} | {{RESULT}} | {{TESTER}} |
| Object storage restore | Quarterly | {{DATE}} | {{RESULT}} | {{TESTER}} |
| Full DR failover drill | Bi-annually | {{DATE}} | {{RESULT}} | {{TESTER}} |

---

## 5. Failover Procedures

### 5.1 Automated Failover

<!-- GUIDANCE: Document what fails over automatically without human intervention. -->

| Component | Automatic Failover | Mechanism | Failover Time |
|-----------|--------------------|-----------|---------------|
| Database (Multi-AZ) | Yes | RDS automatic failover | 60-120 seconds |
| Load balancer | Yes | Health check → route to healthy targets | < 30 seconds |
| CDN | Yes | Origin health checks | < 60 seconds |
| Redis (if clustered) | Yes | Redis Sentinel / ElastiCache | < 30 seconds |

**Monitoring automatic failover:**
- Alert fires: `MultiAZFailover` CloudWatch event or equivalent
- On-call notified immediately
- No manual action required, but on-call must confirm recovery

### 5.2 Manual Failover Steps

<!-- GUIDANCE: Document step-by-step manual failover. Written so anyone on the on-call team can execute it. -->

**Prerequisite:** Automatic failover has NOT occurred or has failed.

#### Database Manual Failover (Tier 1)

1. Confirm primary is unavailable: `ping {{DB_PRIMARY_HOST}}` — should timeout
2. Connect to standby: `psql {{STANDBY_HOST}}`
3. Promote standby to primary: `SELECT pg_promote();`
4. Update DNS record `db.{{INTERNAL_DOMAIN}}` → `{{STANDBY_HOST}}`
5. DNS TTL: Ensure TTL was set to 60s pre-incident (if not, wait {{DNS_TTL}} seconds)
6. Verify applications are reconnecting: Check application logs for successful DB connections
7. Page on-call to verify all services healthy

#### Regional Failover (Catastrophic)

1. Declare DR event (approval from {{DR_AUTHORITY}})
2. Confirm primary region {{PRIMARY_REGION}} is unreachable
3. Activate standby in {{DR_REGION}}: `terraform apply -var-file=envs/dr.tfvars`
4. Restore database from latest cross-region snapshot
5. Update Route 53 / DNS to point to {{DR_REGION}} endpoints
6. Run smoke tests: `bash scripts/smoke-tests.sh {{DR_REGION}}`
7. Notify stakeholders (see Communication Plan)
8. Monitor enhanced metrics for {{MONITOR_PERIOD}}h

---

## 6. Recovery Procedures Per Service

<!-- GUIDANCE: Specific recovery steps for each service tier. Reference runbooks for detail. -->

### Tier 1 Services

| Service | Recovery Procedure | Recovery Script | Est. Time |
|---------|-------------------|-----------------|-----------|
| {{SERVICE_1}} | 1. Restore from snapshot<br/>2. Verify config<br/>3. Run smoke tests | `scripts/restore-{{SERVICE_1}}.sh` | {{TIME}}min |
| Authentication | 1. Deploy from last known good image<br/>2. Verify JWT keys<br/>3. Test login flow | `scripts/restore-auth.sh` | {{TIME}}min |

### Tier 2 Services

<!-- TODO: Document Tier 2 recovery procedures -->

### Tier 3 Services

<!-- TODO: Document Tier 3 recovery procedures -->

---

## 7. DR Drill Schedule & Scenarios

<!-- GUIDANCE: Drills must be realistic. Tabletop exercises are good; real failovers are better. -->

| Drill Type | Frequency | Participants | Last Executed | Next Scheduled |
|------------|-----------|-------------|---------------|----------------|
| Tabletop exercise | Quarterly | On-call team + engineering lead | {{DATE}} | {{DATE}} |
| Database failover test | Quarterly | DevOps + one developer | {{DATE}} | {{DATE}} |
| Full DR failover | Bi-annually | Entire engineering team | {{DATE}} | {{DATE}} |
| Backup restore test | Monthly | DevOps | {{DATE}} | {{DATE}} |

**Drill Scenarios to Cover:**
1. Database primary failure (automatic failover test)
2. Accidental data deletion (point-in-time restore)
3. Single AZ outage (multi-AZ failover)
4. Full region failure (cross-region DR)
5. Ransomware/data corruption (restore from offline backup)
6. CDN outage (origin fallback)
7. Secret store unavailable (cached credentials)

---

## 8. Communication Plan During DR Event

<!-- GUIDANCE: Define who communicates what, to whom, and at what frequency during a DR event. -->

### Internal Communications

| Audience | Channel | Frequency | Owner |
|----------|---------|-----------|-------|
| Engineering team | Slack #incidents + war room call | Real-time | Incident commander |
| Engineering management | Direct message | At declaration + hourly | Incident commander |
| Product/Business leadership | Email + Slack | At declaration + hourly | Incident commander |
| Customer support | Dedicated Slack channel | At declaration + 30 min | Support lead |

### External Communications

| Audience | Channel | Trigger | Message |
|----------|---------|---------|---------|
| Customers | Status page ({{STATUS_PAGE}}) | Within 15 min of confirmed incident | "We are investigating an issue" |
| Customers | Status page update | Every 30 min | Progress update |
| Customers | Email | If impact > {{EMAIL_THRESHOLD}}h | Direct notification |
| SLA customers | Direct contact | Per SLA contract | As contractually required |

**Communication templates:** See [go-live-runbook.md](./go-live-runbook.md) communication section

---

## 9. War Room Setup

<!-- GUIDANCE: Define the virtual/physical command center for managing a DR event. -->

**War Room:** {{WAR_ROOM_LINK}} <!-- Zoom link / Slack channel / Google Meet -->
**Bridge Line:** {{BRIDGE_NUMBER}} <!-- Always-available phone bridge -->
**Document:** Live incident doc created at: {{INCIDENT_DOC_TEMPLATE}}

**Roles during DR event:**

| Role | Responsibility | Primary | Backup |
|------|---------------|---------|--------|
| Incident Commander | Coordinates response, final decisions | {{IC}} | {{IC_BACKUP}} |
| Technical Lead | Leads technical recovery | {{TECH_LEAD}} | {{TECH_BACKUP}} |
| Communications Lead | Internal/external updates | {{COMMS_LEAD}} | {{COMMS_BACKUP}} |
| Scribe | Documents timeline, actions taken | {{SCRIBE}} | Rotate |

---

## 10. Post-Recovery Verification Checklist

<!-- GUIDANCE: After recovery, verify the system is fully functional before standing down. -->

- [ ] All Tier 1 services healthy (health checks passing)
- [ ] Error rate back to baseline (< {{ERROR_BASELINE}}%)
- [ ] P99 latency back to baseline (< {{P99_BASELINE}}ms)
- [ ] Database connections stable
- [ ] Replication lag < {{REPLICATION_LAG}}s (if applicable)
- [ ] Backup jobs resumed and completed successfully
- [ ] Monitoring and alerting functional
- [ ] No data loss confirmed (or data loss quantified and documented)
- [ ] All Tier 2 services healthy
- [ ] Stakeholders notified of recovery
- [ ] Status page updated to "Resolved"
- [ ] Incident timeline documented
- [ ] Post-mortem scheduled (within {{POSTMORTEM_SLA}}h)

---

## 11. DR Test Results Log

<!-- GUIDANCE: Record every test to track improvement over time and satisfy compliance requirements. -->

| Date | Test Type | Scenario | RTO Achieved | RPO Achieved | Issues Found | Resolved By |
|------|-----------|----------|-------------|-------------|--------------|-------------|
| {{DATE}} | {{TYPE}} | {{SCENARIO}} | {{RTO}} | {{RPO}} | {{ISSUES}} | {{RESOLVED}} |

<!-- TODO: Add test results as drills are conducted -->

---

## Related Documents

- [Monitoring & Observability](./monitoring-observability.md)
- [Operational Runbook](../OPERATIONS/operational-runbook.md)
- [Incident Report](../OPERATIONS/incident-report.md)
- [Post-Mortem](../OPERATIONS/post-mortem.md)

---

## Approval
| Role | Name | Date | Signature |
|------|------|------|-----------|
| Author | | | |
| Reviewer | | | |
| Approver | | | |