Skip to main content

ALAI Client Incident Response — Runbook

ALAI Client Incident Response — Runbook

Purpose: What to do when a client-facing incident happens — classify, respond, communicate, escalate. This is the condensed operational runbook; the full process (RCA methodology, post-mortem template, GDPR breach templates) lives in ~/ALAI/processes/incident-management.md. For diagnosing ALAI's own infrastructure alerts (tunnel/daemon/Docker), see ~/system/docs/runbooks/incident-response-playbook.md. Audience: John (primary), Alem (P1 escalation), FlowForge/CodeCraft/Securion (delegated fixes) Last updated: 2026-07-28

Step 1 — Classify Severity Immediately

Severity Description Response Time Resolution Target Who's Notified
P1 CRITICAL Service completely down, active data breach, revenue impact, legal/regulatory violation 15 min 4 hours Alem + Client + Datatilsynet (if GDPR)
P2 HIGH Major feature broken, client blocked, significant perf degradation, security vulnerability 1 hour 8 hours John + Tech Lead + Client
P3 MEDIUM Minor feature broken, workaround exists, non-critical service impaired 4 hours 24 hours John + relevant agent
P4 LOW Cosmetic issue, no business impact 24 hours 5 days Assigned agent

When in doubt, escalate up — better to downgrade a P1 to P2 than to underestimate a critical incident.

P1 triggers include: any production service unavailable to users, data loss/corruption, active breach, GDPR breach (72h Datatilsynet deadline), financial system down (invoicing/payment), contractual SLA breach.

Step 2 — Triage & Open the Incident Record

Owner: John, within the response-time window above.

  1. Acknowledge the reporter (client email, Mattermost, monitoring alert).
  2. Classify severity (Step 1).
  3. Assign an owner — route to the right specialist company (see ~/system/agents/specialist-mapping.json; infra → FlowForge, security → Securion, backend bug → CodeCraft).
  4. Create the incident record from ~/ALAI/processes/templates/OPERATIONS/incident-report.md, ID format INC-YYYY-MM-DD-NNN.
  5. Open a Mattermost thread in #incidents.
node ~/system/tools/mc.js add "INC-<date>-<n>: <summary>" --priority H --owner john

Step 3 — Investigate (timeboxed by severity)

  • P1/P2: fix plan within 1h (P1) / 4h (P2), even before root cause is fully known — mitigate in parallel.
  • P1/P2: mandatory detailed RCA (5 Whys / Fishbone / Timeline). P3: RCA only if recurring or security-related. P4: brief root-cause note.
  • Gather evidence: logs, metrics, screenshots, curl/docker output — verify by live evidence, not by assumption (ZAKON NULA — no claim without tool-verified evidence).

If this is an ALAI infrastructure alert (tunnel down, daemon crashed, Docker container dead) rather than a client-reported bug, go straight to the Alert Triage Matrix in incident-response-playbook.md — it has the exact diagnostic commands (curl -I, docker ps, launchctl list, etc.) per alert type.

Step 4 — Resolve & Verify

  • Deploy fix per normal deploy protocol — ZAKON PI2 applies: no deploy without the 6 hard checks (pre-flight curl + git log + CI health, post-deploy curl 200 + screenshot + new revision serving 100%).
  • Verification window: 1–24h post-fix depending on severity — confirm the client-visible symptom is actually gone (curl 200 != works — check the real workflow).

Step 5 — Communicate with the Client

Use the templates in ~/ALAI/processes/incident-management.md §6:

  • 6.4 Initial notification — sent as soon as a P1/P2 is confirmed and classified.
  • 6.5 Progress update — sent per severity cadence (P1: hourly or on material change; P2: every 4h).
  • 6.6 Resolution confirmation — sent once verification (Step 4) is complete.
  • 6.7 / 6.8 GDPR breach + data-subject notification — mandatory if personal data was exposed; 72h Datatilsynet deadline starts at detection, not confirmation.

Never send a client communication without Alem's awareness on P1 — phone first, Slack second, per the escalation table below.

Step 6 — Escalation Path

Incident Type Escalate To When Method
Public surface down > 5 min Alem Immediately Slack DM + Phone +47 404 74 251
Revenue event (payment failing) Alem Immediately Phone first, Slack second
Security breach / suspicious activity Alem + Securion Immediately Slack #ops + Email [email protected]
GDPR / legal / regulatory issue Alem Within 1 hour Phone + Email
Client contractual SLA breach Alem Immediately Phone + Email
Builder agent repeated failures (3+/hour) Petter Graff Within 1 hour Slack #ops → delegate fix
Non-critical daemon down John (self-fix) Within 15 min Investigate + restart

If John is offline, all P0/P1 alerts route to Alem via email ([email protected]).

Step 7 — Post-Mortem & Closure

  • P1/P2: post-mortem required, using the template in ~/ALAI/processes/incident-management.md §7 (or ~/ALAI/processes/templates/OPERATIONS/post-mortem.md). P3: optional.
  • Closure requires: fix verified live, client notified (if applicable), post-mortem filed, lessons logged.
  • Log the lesson: discover.js memory entry or feedback memory if the root cause reveals a repeatable gap — don't let the same incident recur silently.

Hard Rules

  1. Classify before you act — no fix attempt without a severity label; the label sets the response clock.
  2. No claim without evidence — verify fixes with live curl/screenshot/enumeration, not a green build (ZAKON NULA).
  3. No deploy without ZAKON PI2's 6 checks, even mid-incident.
  4. P1 always reaches Alem — phone first. Never sit on a P1 to "confirm root cause first."
  5. GDPR clock starts at detection — 72h to Datatilsynet, not at RCA completion.
  6. Client comms use the approved templates — don't freelance the wording on data-loss or breach notices; legal risk.
  7. Every P1/P2 gets a post-mortem — no exceptions, no "fixed already, skip write-up."

References

  • Full incident process (classification detail, RCA methodology, all 8 communication templates, GDPR forms): ~/ALAI/processes/incident-management.md
  • Infra alert diagnostics (tunnel/daemon/Docker triage matrix): ~/system/docs/runbooks/incident-response-playbook.md
  • Incident report template: ~/ALAI/processes/templates/OPERATIONS/incident-report.md
  • Post-mortem template: ~/ALAI/processes/templates/OPERATIONS/post-mortem.md
  • SLA report template: ~/ALAI/processes/templates/OPERATIONS/sla-report.md
Back to top