ALAI Backup Strategy

ALAI Backup Strategy

⚠️ STALE-DOC BANNER (2026-08-08, MC #106946): the Layer 3 block below (SSH to Azure VM vm-alai-lightrag, 4 bare volume names) is stale — since the 2026-08-03 migration (#106747) the backup runs entirely local on this Mac against 7 lr106747-*-prefixed volumes, no SSH. Corrected, current detail: LightRAG Backup runbook (updated 2026-08-08). Not fully rewriting this page — see MC #106946 evidence for what changed and why.

Status: LIVE — verified against running processes, crontab, launchctl and logs on 2026-07-28. Owner: FlowForge (infra) / John (orchestration) Supersedes for accuracy: ~/system/architecture/backup-strategy-2026-04-20.md (that document is a PLAN dated 2026-04-20; several of its proposals were never activated or were later reversed by CEO decision — see §5 "Divergence from the original plan"). This page reflects what is actually running today.

1. Architecture — 4 layers, verified live

ANVIL (Mac Studio, 100.103.49.98)
│
├── Layer 1 — Git (hourly)
│     tools/hourly-backup.sh — cron 0 * * * *
│     Commits + pushes ~/system, ~/ALAI, ~/.claude, and every repo under ~/projects/*/*
│     to its GitHub/origin remote. Silent-fail-safe: logs push failures, retries next hour.
│
├── Layer 2 — SQLite (continuous + daily + 2x offsite)
│     PRIMARY: litestream replicate daemon (LaunchAgent com.alai.litestream, PID confirmed
│       live 2026-07-28, running since 2026-07-14). Config: ~/system/config/litestream.yml.
│       64 databases, WAL-shipping, sync interval 1s–300s depending on P0/P1/P2 tier.
│       ALL 64 replicas are `type: file` (LOCAL) → ~/backups/litestream/<db>/ (1.1 GB local,
│       verified 2026-07-28). ZERO Azure (`type: abs`) replicas remain — see §5.
│     SECONDARY (local snapshot): tools/db-backup.sh — cron 0 3 * * *, 5-day local retention,
│       ~/system/backups/databases/
│     OFFSITE #1: tools/rclone-backup.sh — launchd daily 03:00 (com.john.rclone-backup) →
│       Backblaze B2 (b2-alai:alai-studio-backup/system-databases/ + /claude-memory/).
│       Confirmed live run 2026-07-28T03:00Z, OK.
│     OFFSITE #2: daemons/offsite-backup.sh — launchd every 6h (com.john.offsite-backup) →
│       Backblaze B2 (b2-alai:alai-studio-backup/databases/, /config/, /rules/, /specs/,
│       /tools/, /claude-hooks/, /claude-agents/, /claude-skills/, /claude/).
│       Confirmed live run 2026-07-28T03:36Z, 9/9 targets OK, 85s.
│
├── Layer 3 — LightRAG / Neo4j Docker volumes
│     tools/lightrag-backup.sh — launchd weekly Sunday 04:00 (com.alai.lightrag-backup)
│     SSH to Azure VM vm-alai-lightrag (20.240.61.67) → docker compose stop → tar each of
│       4 volumes (lightrag-data, lightrag-kg, lightrag-cache, lightrag-neo4j-data) →
│       docker compose start → SHA-256 manifest.
│     SCP snapshot back to Mac Studio (~/system/backups/lightrag/, local safety net, keep 4).
│     Azure offsite upload → Storage Account `plockfrontstaging` (swedencentral),
│       Container `lightrag-backup`, Cool tier, keep last 8 snapshots (~8 weeks).
│     Full detail: [LightRAG Backup runbook](./lightrag-backup.md)
│
└── Layer 4 — Full ANVIL loss
      No fully-automated one-shot bootstrap is active today. Recovery = re-run Layers 1-3
      restore procedures on a freshly provisioned Mac. See Disaster Recovery Runbook.

2. Azure storage inventory (tool-verified 2026-07-28)

3. Retention policy (as configured, verified)

The April plan's proposed 30-day-Cool → Archive → 365-day-delete lifecycle on alaibackups0ebb applies to the SQLite/git-bundle layer that was never activated in production — treat that lifecycle description as design intent, not live policy.

4. Cost (verified against live config, not the April estimate)

• SQLite + git layer: effectively $0 in direct Azure spend today — litestream runs against local file replicas, git backup is a GitHub push, B2 offsite is Backblaze (separate low-cost provider, not itemized this session).
• LightRAG/Neo4j layer: per lightrag-backup.md, ~$0.05–0.10/month on plockfrontstaging Cool tier for ~4 GB retained.
• The April plan's ~$2–4/month estimate was for a dedicated alaibackups<suffix> account running the full SQLite+git Azure Blob lifecycle. That path is not the one in production, so that estimate does not describe current spend.

5. Divergence from the original 2026-04-20 plan — why, and the authority for it

CEO decision, verbatim, 2026-07-13 (MC #105462): "Prebaci lokalno! Nemamo placene korisnika kad to dodje ide backup na azure do tada save money gdje mozemo." ("Move it local! We don't have paying customers — when that happens, backup goes to Azure, until then save money wherever we can.")

Context: MC #105462 — litestream's per-sync-tick ListBlobs full-prefix enumeration against alaibackups0ebb (two hot telemetry DBs, flywheel/mission-control, at ~113K/~102K blobs each) was driving up to 518 GB/day list-egress (~370 NOK/day). After phased mitigation attempts (sync-interval tuning, lifecycle rules) proved insufficient, the CEO's second, later, better-informed decision (received ~10 hours after an earlier, more conservative one — the later decision took precedence per session record) was to migrate all litestream replicas to local file targets rather than keep paying Azure list-egress with no paying customers yet. Full incident record: Azure litestream egress saga + backup policy — MC #105462.

Practical effect: the April plan's Layer 2 (SQLite → Azure Blob, dedicated alaibackups0ebb account) is stood up in config/scripts but intentionally not the live path. B2 (Backblaze) offsite + local litestream replicas carry that layer today. LightRAG/Neo4j (Layer 3) was unaffected — it is a low-frequency (weekly), low-blob-count workload on a different storage account, so it kept its original Azure design.

IAM note: the SP configured for the Azure Blob SQLite path (alai-backup-writer / appid 1a0b3018-…, per ~/system/config/litestream.yml and azure-backup.env) is the same appid used as the general-purpose az-cli identity elsewhere in the system. VAULT_EXPORTER_APPID in azure-backup.env is still FILL_AFTER_SP_CREATION — that narrower-scoped SP for Bitwarden exports was never created. Do not assume it exists.

6. Reactivation trigger

Per the CEO decision above, the Azure Blob SQLite backup path (alaibackups0ebb) should be reactivated when ALAI has paying customers. Reactivation requires, at minimum: (1) re-verify RBAC on alai-backups-rg/alaibackups0ebb (current az-cli SP identity failed this check on 2026-07-28), (2) re-enable com.alai.azure-db-backup.plist (currently .disabled), (3) re-point litestream replicas for at least the P0-financial tier back to type: abs, (4) re-run the lifecycle-policy fix from #105490 (an account-wide lifecycle rule without prefixMatch archived 35 dormant DBs' L0 blobs during the original incident — must be scoped before reactivation).

Document Owner: Lexicon (Skillforge role) Last Verified: 2026-07-28 — process list (litestream replicate, live PID), crontab -l, launchctl list, LaunchAgent plists, live log tails (offsite-backup.log, rclone-backup-20260728.log), az account show / az storage account show (AuthorizationFailed, documented as-is), and ~/system/config/litestream.yml (64/64 DBs confirmed type: file).