# Operational Runbook

# Operational Runbook

> **Project:** {{PROJECT_NAME}}
> **Version:** {{VERSION}}
> **Date:** {{DATE}}
> **Author:** {{AUTHOR}}
> **Status:** Draft | In Review | Approved
> **Reviewers:** {{REVIEWERS}}

## Document History
| Version | Date | Author | Changes |
|---------|------|--------|---------|
| 0.1     | {{DATE}} | {{AUTHOR}} | Initial draft |

---

## 1. Service Overview

<!-- GUIDANCE: Brief description of the system for on-call engineers who may not be familiar with the service. -->

**Service:** {{PROJECT_NAME}}
**Purpose:** {{SERVICE_PURPOSE}}
**Technology stack:** {{STACK}} <!-- e.g., "Node.js + PostgreSQL + Redis, deployed on AWS ECS" -->
**Architecture reference:** [Deployment Architecture](../INFRASTRUCTURE/deployment-architecture.md)

**Service URLs:**

| Environment | URL | Health Check |
|-------------|-----|--------------|
| Production | `{{PROD_URL}}` | `{{PROD_URL}}/health` |
| Staging | `{{STG_URL}}` | `{{STG_URL}}/health` |

**Key dashboards:**
- System overview: {{DASHBOARD_LINK}}
- Service metrics: {{SERVICE_DASHBOARD_LINK}}
- Logs: {{LOG_DASHBOARD_LINK}}

---

## 2. Common Operational Tasks

### 2.1 Service Restart Procedure

<!-- GUIDANCE: Write this so an on-call engineer can execute it at 3am without knowing the service intimately. -->

**When to use:** Application unresponsive, hanging workers, suspected deadlock

**Steps:**

**Option A — Rolling restart (no downtime):**
```bash
# AWS ECS
aws ecs update-service --cluster {{CLUSTER}} --service {{SERVICE}} --force-new-deployment

# Kubernetes
kubectl rollout restart deployment/{{DEPLOYMENT}} -n {{NAMESPACE}}
```

**Option B — Emergency restart (brief downtime, use only if rolling restart fails):**
```bash
# Stop all instances
{{STOP_COMMAND}}
# Wait for drain
sleep 30
# Start fresh
{{START_COMMAND}}
```

**Verify:**
```bash
# Check all instances healthy
{{HEALTH_CHECK_COMMAND}}
# Check for errors post-restart
{{LOG_CHECK_COMMAND}}
```

**Expected restart time:** {{RESTART_TIME}} minutes
**Alert expected:** Service restart will trigger deployment alert — acknowledge in PagerDuty

---

### 2.2 Log Retrieval & Analysis

<!-- GUIDANCE: Document how to quickly get to relevant logs. Time matters during incidents. -->

**Centralized logs:** {{LOG_URL}}

**Quick log retrieval:**
```bash
# Last 100 error lines
{{LOG_TOOL}} --filter "level=error" --since "1h" --service {{SERVICE}}

# Logs for a specific user
{{LOG_TOOL}} --filter "user_id={{USER_ID}}" --since "24h"

# Logs for a specific request
{{LOG_TOOL}} --filter "request_id={{REQUEST_ID}}"

# Database slow query logs
{{DB_LOG_COMMAND}}
```

**Log format reference:** See [Monitoring & Observability](../INFRASTRUCTURE/monitoring-observability.md#structured-logging-format)

---

### 2.3 Database Maintenance

**Connection count check:**
```sql
SELECT count(*) as connections, state FROM pg_stat_activity GROUP BY state;
```

**Kill idle connections:**
```sql
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE state = 'idle'
  AND state_change < now() - interval '5 minutes'
  AND pid <> pg_backend_pid();
```

**Running queries (detect long-running):**
```sql
SELECT pid, duration, query, state
FROM pg_stat_activity
WHERE (now() - pg_stat_activity.query_start) > interval '1 minute'
  AND state != 'idle';
```

**Vacuum / analyze (if table bloat suspected):**
```sql
VACUUM ANALYZE {{TABLE_NAME}};
```

**Check replication lag:**
```sql
SELECT now() - pg_last_xact_replay_timestamp() AS replication_lag;
```

---

### 2.4 Cache Clearing / Warming

**Clear all cache (use with caution — may spike DB load):**
```bash
{{CACHE_FLUSH_COMMAND}}
```

**Clear specific key pattern:**
```bash
{{CACHE_DELETE_PATTERN_COMMAND}}
```

**Check cache hit rate:**
```bash
{{CACHE_STATS_COMMAND}}
```

**Warm cache after clearing:**
```bash
# Run cache warming script
bash scripts/warm-cache.sh {{ENVIRONMENT}}
# Or trigger warming job
{{WARM_CACHE_JOB_COMMAND}}
```

**Expected DB load spike after cache clear:** {{CACHE_CLEAR_IMPACT}} minutes of elevated load

---

### 2.5 Certificate Renewal

<!-- GUIDANCE: Certificates should renew automatically. This covers the manual process for when automation fails. -->

**Automated renewal:** Configured via {{CERT_TOOL}} (Let's Encrypt / ACM)
**Auto-renewal trigger:** 30 days before expiry

**Manual renewal (if auto-renewal fails):**
```bash
# Check expiry
echo | openssl s_client -connect {{DOMAIN}}:443 2>/dev/null | openssl x509 -noout -dates

# Manual renewal
{{CERT_RENEW_COMMAND}}

# Verify
{{CERT_VERIFY_COMMAND}}
```

**Verify renewal alert is working:**
- Alert configured: "Certificate expiring in < 30 days" → {{ALERT_CHANNEL}}
- Test certificate: `curl -I https://{{DOMAIN}}` and check `Strict-Transport-Security` header

---

### 2.6 Scaling Up / Down

**Scale up (increase capacity):**
```bash
# AWS ECS
aws ecs update-service --cluster {{CLUSTER}} --service {{SERVICE}} --desired-count {{COUNT}}

# Kubernetes
kubectl scale deployment/{{DEPLOYMENT}} --replicas={{COUNT}} -n {{NAMESPACE}}
```

**Verify scale-out:**
```bash
# Check instance count
{{INSTANCE_COUNT_COMMAND}}
# Confirm health
{{HEALTH_CHECK_COMMAND}}
```

**Scale down (reduce capacity — use cautiously):**
- Do NOT scale below {{MIN_INSTANCES}} instances
- Scale down during off-peak hours only ({{OFF_PEAK_HOURS}})
- Monitor for 10 minutes after scaling down to confirm stability

---

## 3. Troubleshooting Playbooks

### 3.1 High CPU Usage

<!-- GUIDANCE: Systematic approach to CPU issues. Don't panic, follow the steps. -->

**Symptoms:** CPU alert fires, slow responses, possible OOM

1. **Identify the source:**
   ```bash
   # Top processes by CPU
   {{CPU_TOP_COMMAND}}
   ```
2. Check for: runaway loops, large queries being processed, missing cache causing recalculation
3. **Check for recently deployed code** — did CPU spike after a deploy? → Consider rollback
4. **Check queue depth** — backed-up job queue causes worker CPU spike
5. If single instance: restart that instance (`{{RESTART_SINGLE_COMMAND}}`)
6. If all instances: scale up immediately, then investigate root cause
7. **Escalate if:** CPU > {{CPU_ESCALATE}}% for > {{ESCALATE_DURATION}} min after scaling

---

### 3.2 Memory Leaks

**Symptoms:** Slowly increasing memory, eventual OOM kill / restart loop

1. **Check memory trend** in monitoring dashboard — linear increase over hours = leak
2. **Identify the leak:**
   - Enable heap dump: `{{HEAP_DUMP_COMMAND}}`
   - Profile with: `{{PROFILER}}`
3. **Short-term mitigation:** Schedule rolling restarts every {{RESTART_INTERVAL}}h
   ```bash
   {{SCHEDULED_RESTART_COMMAND}}
   ```
4. **Create ticket** with heap dump attached — requires developer investigation
5. **Escalate if:** Restart cycle < {{MIN_RESTART_INTERVAL}}h (memory fills too fast)

---

### 3.3 Slow Database Queries

**Symptoms:** High P99 latency, DB CPU spike, timeouts in logs

1. **Find slow queries:**
   ```sql
   SELECT query, calls, mean_exec_time, max_exec_time
   FROM pg_stat_statements
   ORDER BY mean_exec_time DESC
   LIMIT 20;
   ```
2. **Check for missing indexes:** Look for sequential scans on large tables
3. **Check for blocking queries:**
   ```sql
   SELECT blocking.pid, blocking.query, blocked.pid, blocked.query
   FROM pg_stat_activity blocked
   JOIN pg_stat_activity blocking ON blocking.pid = ANY(pg_blocking_pids(blocked.pid));
   ```
4. **Kill blocking query if safe:**
   ```sql
   SELECT pg_cancel_backend({{PID}});
   -- If cancel doesn't work:
   SELECT pg_terminate_backend({{PID}});
   ```
5. **Create ticket** — developer must optimize the query

---

### 3.4 Service Connectivity Issues

**Symptoms:** Connectivity errors between services, 502/503 errors

1. **Check health endpoints:**
   ```bash
   curl -I {{SERVICE_URL}}/health
   ```
2. **Check network security groups / firewall rules** — was anything changed recently?
3. **Check service discovery** — DNS resolving correctly?
   ```bash
   nslookup {{SERVICE_INTERNAL_DNS}}
   ```
4. **Check if service is running:**
   ```bash
   {{SERVICE_STATUS_COMMAND}}
   ```
5. **Check logs for connection errors:**
   ```bash
   {{CONNECTIVITY_LOG_COMMAND}}
   ```

---

### 3.5 High Error Rates

**Symptoms:** Error rate alert, user complaints, 5xx in logs

1. **Identify error type:** `{{LOG_ERROR_COMMAND}}` — what errors, what services, what endpoints?
2. **Check if correlated with:** recent deployment, external service outage, traffic spike
3. **Check external service status pages:**
   - {{SERVICE_1}} status: {{STATUS_PAGE_1}}
   - {{SERVICE_2}} status: {{STATUS_PAGE_2}}
4. **If recent deployment:** Consider rollback if errors affecting > {{ROLLBACK_ERROR_THRESHOLD}}% of requests
5. **If external service down:** Check circuit breaker status, enable fallback
6. **Escalate if:** Error rate > {{ESCALATE_ERROR_RATE}}% for > {{ESCALATE_DURATION}} min

---

### 3.6 Disk Space Issues

**Symptoms:** Disk space alert, application errors writing files

1. **Check disk usage:**
   ```bash
   df -h
   du -sh /var/log/* | sort -rh | head -10
   ```
2. **Quick wins:**
   ```bash
   # Rotate and compress logs
   logrotate -f /etc/logrotate.conf
   # Clear old Docker images
   docker image prune -a --filter "until=24h"
   # Clear /tmp
   find /tmp -mtime +7 -delete
   ```
3. **If database disk:** Check for table bloat, dead tuples, WAL accumulation
   ```sql
   SELECT pg_size_pretty(pg_database_size('{{DB_NAME}}'));
   ```
4. **Escalate if:** Disk > {{DISK_ESCALATE}}% and cannot free space quickly

---

## 4. Health Check Endpoints

<!-- GUIDANCE: Document all health check endpoints and what a healthy response looks like. -->

| Endpoint | Method | Expected Response | What It Checks |
|----------|--------|------------------|----------------|
| `{{BASE_URL}}/health` | GET | HTTP 200 `{"status":"ok"}` | Application running |
| `{{BASE_URL}}/health/ready` | GET | HTTP 200 `{"status":"ready"}` | App + DB + Cache connected |
| `{{BASE_URL}}/health/live` | GET | HTTP 200 `{"status":"alive"}` | App process alive |
| `{{BASE_URL}}/health/db` | GET | HTTP 200 `{"status":"ok","latency_ms":X}` | Database reachable |
| `{{BASE_URL}}/health/cache` | GET | HTTP 200 `{"status":"ok"}` | Redis reachable |

**Health check from load balancer:** `{{HEALTH_CHECK_PATH}}` every {{LB_INTERVAL}}s
**Unhealthy threshold:** {{UNHEALTHY_COUNT}} consecutive failures

---

## 5. Alert Response Procedures

<!-- GUIDANCE: For each alert, document the immediate response steps. Don't make on-call guess. -->

| Alert | Immediate Action | Runbook Section |
|-------|-----------------|-----------------|
| `HighErrorRate` | Check logs, identify error type, assess scope | 3.5 High Error Rates |
| `SlowP99` | Check DB slow queries, recent deploys | 3.3 Slow DB Queries |
| `ServiceDown` | Restart service, check logs | 2.1 Service Restart |
| `HighCPU` | Scale up, identify source | 3.1 High CPU |
| `DiskAlmostFull` | Clear logs/tmp, escalate if > 90% | 3.6 Disk Space |
| `DBReplicationLag` | Check replication, network, disk on replica | DB section |
| `CertificateExpiring` | Trigger manual renewal | 2.5 Certificate Renewal |

---

## 6. Escalation Matrix

<!-- GUIDANCE: Who to call when you can't solve it yourself. Names, not just roles. -->

| Situation | First Contact | Escalation | Ultimate Escalation |
|-----------|--------------|------------|---------------------|
| Service down | On-call engineer | Tech lead | Engineering manager |
| Data loss / corruption | On-call + Tech lead | CTO | CTO |
| Security incident | Security contact | CISO | CEO |
| Payment system down | On-call + Payment owner | Stripe/payment provider support | Engineering manager |

**Emergency contacts:**
| Role | Name | Phone | Slack |
|------|------|-------|-------|
| On-call (primary) | {{PRIMARY}} | {{PHONE}} | {{SLACK}} |
| On-call (backup) | {{BACKUP}} | {{PHONE}} | {{SLACK}} |
| Tech Lead | {{TECH_LEAD}} | {{PHONE}} | {{SLACK}} |
| Engineering Manager | {{ENG_MGR}} | {{PHONE}} | {{SLACK}} |

---

## 7. On-Call Handoff Procedure

<!-- GUIDANCE: A good handoff prevents issues from falling through the cracks. -->

**Handoff cadence:** {{HANDOFF_CADENCE}} <!-- Weekly / every 3 days -->
**Handoff time:** {{HANDOFF_TIME}}

**Outgoing on-call must document:**
- [ ] Any open incidents or ongoing issues
- [ ] Any monitoring anomalies (elevated error rates, slow queries not yet resolved)
- [ ] Any upcoming events that may affect the system (marketing campaigns, scheduled maintenance)
- [ ] Any temporary mitigations in place that need permanent fixes
- [ ] Context on any unusual alerts that fired and were noise

**Handoff document template:** {{HANDOFF_TEMPLATE_LINK}}

---

## 8. Maintenance Window Procedure

<!-- GUIDANCE: Planned maintenance requires coordination. Follow this process to minimize impact. -->

**Maintenance window schedule:** {{MAINTENANCE_WINDOW}} (lowest traffic period)

**Pre-maintenance:**
1. Announce in Slack #ops: "Maintenance window {{DATE}} {{TIME}}-{{END_TIME}}"
2. Update status page: "Scheduled maintenance" with details
3. Notify impacted customers if downtime expected > {{DOWNTIME_NOTIFY_THRESHOLD}} minutes
4. Confirm rollback plan is ready

**During maintenance:**
1. Enable maintenance mode (if applicable): `{{MAINTENANCE_MODE_CMD}}`
2. Execute maintenance tasks per the specific runbook for the task
3. Run smoke tests after each major step
4. Document every action taken with timestamps

**Post-maintenance:**
1. Disable maintenance mode: `{{DISABLE_MAINTENANCE_CMD}}`
2. Run full smoke test suite
3. Monitor for 30 minutes
4. Update status page: "Maintenance complete, all systems normal"
5. Post-maintenance report in #ops Slack channel

---

## Related Documents

- [Go-Live Runbook](./go-live-runbook.md)
- [Incident Report](./incident-report.md)
- [Monitoring & Observability](../INFRASTRUCTURE/monitoring-observability.md)
- [Disaster Recovery Plan](../INFRASTRUCTURE/disaster-recovery-plan.md)

---

## Approval
| Role | Name | Date | Signature |
|------|------|------|-----------|
| Author | | | |
| Reviewer | | | |
| Approver | | | |