Skip to main content

Azure Blob Offsite Backup Setup

Azure Blob Offsite Backup Setup

Status: LIVE — verified against running processes, LaunchAgent plists and config on 2026-07-28. Owner: FlowForge (infra) / John (orchestration) Scope note: This page documents specifically the Azure Blob offsite paths. For the full 4-layer ALAI backup picture (including Git and Backblaze B2), see ALAI Backup Strategy.

1. There are two independent Azure Blob backup paths — one active, one scaffolded

Path Storage account Status
LightRAG / Neo4j Docker volumes plockfrontstaging (swedencentral) ACTIVE — weekly
SQLite databases (mission-control, hivemind, costs, events, etc.) alaibackups0ebb (RG alai-backups-rg) Scaffolded, disabled — see §3

These are unrelated Azure storage accounts serving different data, set up at different times. Do not confuse them.

2. ACTIVE: LightRAG/Neo4j → plockfrontstaging

Script: ~/system/tools/lightrag-backup.sh LaunchAgent: com.alai.lightrag-backup.plist — weekly, Sunday 04:00 CEST Credentials: ~/system/config/azure-lightrag-backup.env (mode 0600) — AZURE_STORAGE_ACCOUNT, AZURE_STORAGE_KEY, AZURE_STORAGE_CONTAINER

What happens:

  1. SSH to Azure VM vm-alai-lightrag (20.240.61.67)
  2. docker compose stop lightrag neo4j (graceful, ~30-90s downtime)
  3. Tar each of 4 Docker volumes (lightrag-data, lightrag-kg, lightrag-cache, lightrag-neo4j-data)
  4. docker compose start neo4j lightrag
  5. SHA-256 manifest + README written alongside the snapshot
  6. SCP snapshot back to Mac Studio (~/system/backups/lightrag/, local safety net, keep last 4)
  7. Azure Blob upload — az storage blob upload per file, --tier Cool, blob path <container>/<timestamp>/<file>
  8. Azure-side rotation — keep last 8 snapshots (~8 weeks), older ones deleted via az storage blob delete-batch

Azure resource details:

  • Storage account: plockfrontstaging (swedencentral, Hot-tier account — a pre-existing Plock account being reused for this, not a dedicated ALAI backup account)
  • Container: lightrag-backup
  • Resource group: plock-staging-rg
  • Per-blob access tier: Cool
  • Retention: last 8 snapshots offsite (vs. 4 kept locally)
  • Cost: ~$0.05–0.10/month for ~4 GB retained

Restore: full step-by-step (download-batch, checksum verify, volume restore to Azure VM or to Mac Studio) is documented in LightRAG Backup runbook — do not duplicate here, that page is the source of truth for restore procedure.

3. SCAFFOLDED / DISABLED: SQLite → alaibackups0ebb

This path exists in config and credentials but is intentionally not the live path as of 2026-07-28.

Storage account: alaibackups0ebb, resource group alai-backups-rg Containers (per original design):

Container Purpose
system-db-backups SQLite database blobs (litestream type: abs replicas)
system-git-bundles Git bundle exports
bitwarden-exports Vaultwarden/Bitwarden export snapshots

Credentials: ~/system/config/azure-backup.env (mode 600)

  • AZURE_BACKUP_SP_APPID = 1a0b3018-0c31-474b-918f-531b0a29a669 (SP name: alai-backup-writer) — same appid used as the general-purpose az-cli identity elsewhere in the system, not a narrowly-scoped backup-only SP
  • AZURE_BACKUP_SP_TENANT = 3454a03f-20b4-4bda-a116-2293c459aecd
  • AZURE_SUBSCRIPTION_ID = 5b0b4d9b-e677-464e-abf0-5170cbce3b8e
  • VAULT_EXPORTER_APPID / VAULT_EXPORTER_SECRET = still FILL_AFTER_SP_CREATION — the narrower-scoped SP intended for the bitwarden-exports container only was never created. Do not assume it exists.

LaunchAgent: com.alai.azure-db-backup.plist.disabled (file dated 2026-04-20) — filename literally carries .disabled, RunAtLoad=false, never renamed to an active .plist. Confirmed present but inert on disk 2026-07-28.

Litestream (~/system/config/litestream.yml): documents alai-backup-writer as the intended Azure auth for streaming SQLite replication, and originally shipped replicas of type: abs (Azure Blob Storage) pointed at system-db-backups. As of 2026-07-28 all 64 configured databases use type: file (local) replicas only — zero type: abs replicas remain.

Why it's off — CEO decision, MC #105462 (2026-07-13)

Litestream's per-sync-tick ListBlobs full-prefix enumeration against alaibackups0ebb (driven mainly by two hot telemetry DBs — flywheel ~113K blobs, mission-control ~102K blobs) drove egress up to 518 GB/day (~370 NOK/day). After sync-interval tuning and lifecycle-rule fixes proved insufficient to bring it under control, CEO decision (verbatim): "Prebaci lokalno! Nemamo placene korisnika kad to dodje ide backup na azure do tada save money gdje mozemo." — move everything to local replicas; Azure Blob SQLite backup returns once ALAI has paying customers. Full incident record: Azure litestream egress saga + backup policy — MC #105462.

Reactivation checklist (when triggered)

  1. Re-verify RBAC on alai-backups-rg / alaibackups0ebb — the current az-cli SP identity returned AuthorizationFailed reading this account/RG as of 2026-07-28; access is not currently confirmed.
  2. Re-enable the LaunchAgent: rename com.alai.azure-db-backup.plist.disabled → com.alai.azure-db-backup.plist, launchctl load.
  3. Re-point at least the P0-financial tier of litestream.yml replicas back to type: abs.
  4. Re-scope the account-wide Azure lifecycle rule fixed under #105490 (it lacked prefixMatch and archived L0 blobs for 35 dormant DBs during the original incident) before turning replication back on.

4. Setup commands (as originally provisioned, for reference)

Creating the general-purpose backup SP (already done — this is what AZURE_BACKUP_SP_APPID above corresponds to):

az ad sp create-for-rbac --name alai-backup-writer --role "Storage Blob Data Contributor" \
  --scopes /subscriptions/5b0b4d9b-e677-464e-abf0-5170cbce3b8e/resourceGroups/alai-backups-rg/providers/Microsoft.Storage/storageAccounts/alaibackups0ebb

Creating the narrower vault-export-only SP (documented, never executed — VAULT_EXPORTER_APPID still unfilled):

az ad sp create-for-rbac --name alai-vault-exporter --role "Storage Blob Data Contributor" \
  --scopes /subscriptions/5b0b4d9b-e677-464e-abf0-5170cbce3b8e/resourceGroups/alai-backups-rg/providers/Microsoft.Storage/storageAccounts/alaibackups0ebb/blobServices/default/containers/bitwarden-exports

Document Owner: Skillforge Last Verified: 2026-07-28 — ~/Library/LaunchAgents/com.alai.azure-db-backup.plist.disabled (present, inert), ~/system/config/litestream.yml (64/64 DBs type: file), ~/system/config/azure-backup.env and azure-lightrag-backup.env (contents read directly), ~/system/tools/lightrag-backup.sh (script read directly), cross-checked against ~/system/docs/runbooks/alai-backup-strategy.md (same-day verification).

Back to top