ALAI Client Incident Response — Runbook ALAI Client Incident Response — Runbook Version: 1.0 Created: 2026-07-28 Owner: John (AI Director) / Alem Basic (CEO) Source process: processes/incident-management.md (full spec), processes/quick-ref/incident-response-quickref.md Related: processes/templates/incident-report.md , processes/communication-protocol.md 1. Purpose Client-facing runbook for handling any incident that affects a client's service: production outages, bugs, security incidents, SLA breaches. This is the condensed operational version — the full governance document (classification rationale, GDPR appendix, KPIs, audit trail) lives in processes/incident-management.md . 2. Severity Matrix Severity Definition Response Time Resolution Target Notify P1 CRITICAL Service completely down, active data breach, revenue impact, legal/regulatory violation 15 min 4 hours Alem + Client + Datatilsynet (if GDPR) P2 HIGH Major feature broken, client blocked, no workaround, security vulnerability 1 hour 8 hours John + Tech Lead + Client P3 MEDIUM Minor feature broken, workaround exists 4 hours 24 hours John + relevant agent P4 LOW Cosmetic, no business impact 24 hours 5 days Assigned agent When in doubt, escalate up. Better to downgrade a P1 to P2 than to underestimate a critical incident. 3. Response Flow Detection -> Triage & Classify -> Investigation -> Resolution -> Verification -> Communication -> Post-Mortem -> Closure Step 1 — Detection Source Action Automated monitoring (Mission Control, health checks) Auto-alert to John Client report (email, ticket) Logged via support-ticket.js , escalated by John Internal discovery (agent/dev) Report to John immediately (Slack #ai-ops) Third-party/vendor Forward to John + Alem Step 2 — Triage & Classify (John, within SLA response time above) Acknowledge receipt to the reporter Classify severity using the matrix in Section 2 Assign owner (agent/role per incident type) Create incident record: comms/incidents/INC-YYYY-MM-DD-NNN.md from processes/templates/incident-report.md Notify stakeholders per Escalation Matrix (Section 4) Open Slack thread in #incidents Step 3 — Investigation Reproduce the issue in the affected environment Gather evidence: logs, metrics, screenshots Root cause analysis (5 Whys / Fishbone / Timeline) — mandatory for P1/P2 Assess impact: users, data, services affected Fix plan: immediate mitigation vs. long-term fix Step 4 — Resolution Implement fix (code, config, infra, or process change) Validate: staging for P2-P4, production hotfix + 1h monitoring for P1 Deploy per deployment protocol Monitor metrics/logs/error rates for 1 hour post-deploy Step 5 — Verification Fix deployed to production Smoke tests passing No related errors in the last hour Monitoring shows normal metrics Client confirms resolution (if client-reported) Incident record updated to "Resolved" Step 6 — Communication cadence Severity Initial Update Progress Updates Resolution Notice P1 Within 15 min Every 30 min Immediately + 24h follow-up P2 Within 1 hour Every 2 hours Within 1h of resolution P3 Within 4 hours Daily if >24h Next business day OK P4 N/A unless client-reported N/A Next sprint review Step 7 — Post-Mortem (mandatory P1/P2, optional P3) Scheduled within 2 business days of resolution, conducted within 5 Blameless, fact-based, action-oriented Every post-mortem produces at least one action item, tracked in Mission Control Template and full 10-section structure: see processes/incident-management.md §7.3 Step 8 — Closure Incident resolved and verified Post-mortem completed (P1/P2) Action items created and assigned Documentation/runbooks updated Client notified (final resolution email) Incident record archived to comms/incidents/archive/YYYY/ 4. Escalation Matrix P4: Developer -> Tech Lead (review) -> Archive P3: Developer -> Tech Lead (notified) -> Archive P2: Tech Lead (owns) -> John (notified) -> Client notified -> Archive -> if SLA breach: Alem involved P1: John coordinates -> Alem notified IMMEDIATELY -> Client notified IMMEDIATELY -> Datatilsynet if GDPR breach (72h) Escalate to Alem when: any P1; P2 affecting a major client or revenue; client relationship risk; legal/regulatory implication (GDPR, lawsuit threat); media/PR risk; financial impact >50,000 NOK; requires external assistance. Authority When Deadline Contact Datatilsynet Personal data breach (GDPR Art. 33) 72h from detection postkasse@datatilsynet.no Client Breach of data processing agreement 24h per DPA Client contact per contract Politiet (Kripos) Criminal activity Immediately 02800, tips@politiet.no NSM Critical infrastructure attack Immediately nsm@nsm.no 5. Client Communication Templates Initial notification (email, within SLA) Use values from the incident record and the client contract/contact record before sending. Subject: Action Required or Informational — Service Incident Notification — {{client_name}} We are writing to inform you of a service incident affecting {{affected_service_or_feature}}. Incident Summary: - What: {{impact_what_users_experience}} - When Detected: {{detected_date_time_with_timezone}} - Current Status: {{investigating_mitigating_or_resolved}} - Impact to Your Users: {{specific_user_impact}} What We Are Doing: {{response_actions_underway}} Next Steps: {{client_expectation_workaround_next_update_time}} Estimated Resolution: {{estimated_resolution_or_investigating}} We will provide updates every {{update_cadence}} until resolved. Resolution confirmation Use values from the incident record and production verification evidence before sending. Subject: RESOLVED: Service Incident — {{client_name}} Incident Summary: - Issue: {{issue_summary}} - Root Cause: {{brief_non_technical_root_cause}} - Resolution: {{what_was_fixed}} - Resolved At / Total Duration: {{resolved_timestamp_and_duration}} Preventive Actions: {{prevention_actions}} Impact to Your Data: {{data_impact_statement}} Full template set (internal declared/update/resolved, GDPR Art. 33/34 regulatory notices): processes/incident-management.md §6. 6. On-Call Primary: John (AI Director) — 24/7 via automated monitoring Human backup: Alem (CEO) — for P1 decisions requiring a human Business hours: Mon-Fri 08:00-18:00 CET; after-hours P1 = John responds immediately + alerts Alem by phone, P2 = John within 1h + email to Alem (reviewed within 4h) Contact priority: Slack #ai-ops -> email alem@alai.no -> phone (P1 only) 7. Tools Tool Purpose Command support-ticket.js Create incident record from client report node ~/system/tools/support-ticket.js create --title "" --severity Mission Control Track incident as task node ~/system/tools/mc.js add " " --priority high Slack Real-time incident channel node ~/system/tools/slack.js send incidents "" HiveMind Log lessons learned node ~/system/agents/hivemind/hivemind.js post john incident "" health-check.js System health monitoring node ~/system/tools/health-check.js --quick auto-fix.js Automated recovery for known issues node ~/system/tools/auto-fix.js Mission Control Dashboard: http://localhost:3030 8. Hard Rules Blameless post-mortems — focus on systems/process, never individuals. Every P1/P2 gets a post-mortem within 5 business days, no exceptions. GDPR 72h clock starts at detection , not at confirmation — when in doubt, start the clock. Client communication is never skipped for P1/P2, even if resolution is fast. No incident closes without an incident record in comms/incidents/ . 9. Related Documents processes/incident-management.md — full 13-section governance document (severity rationale, KPIs, RCA appendix, GDPR breach assessment tree, quarterly review process) processes/quick-ref/incident-response-quickref.md — 1-page field version processes/templates/incident-report.md ALAI-CLIENT-ONBOARDING.md ALAI-CLIENT-INVOICE-CYCLE.md Document Location: ~/ALAI/processes/runbooks/ALAI-CLIENT-INCIDENT-RESPONSE.md Access Control: INTERNAL