Azure Blob Offsite Backup Setup

Overview

~~Purpose:~~Status: ~~Offsite~~LIVE — verified against running processes, LaunchAgent plists and config on 2026-07-28. Owner: FlowForge (infra) / John (orchestration) Scope note: This page documents specifically the Azure Blob offsite paths. For the full 4-layer ALAI backup ~~for~~picture ~~ALAI~~(including ~~system databases~~Git and ~~git~~Backblaze ~~bundles~~
B2), ~~Region:~~see ~~North~~ALAI ~~Europe~~Backup ~~(Dublin)~~Strategy.

1. There are two independent Azure Blob backup paths — geographicone separationactive, fromone primary Sweden Central VM
Retention: 365 days with lifecycle policies (Hot → Cool → Archive → Delete)
Recovery Time Objective: 4 hours (manual restore)

Azure Resourcesscaffolded

~~Type~~

~~Resource~~Path	Storage account	~~Name~~Status	~~Purpose~~
~~Resource~~LightRAG ~~Group~~/ Neo4j Docker volumes	`plockfrontstaging` (swedencentral)	ACTIVE — weekly
SQLite databases (mission-control, hivemind, costs, events, etc.)	`alaibackups0ebb` (RG `alai-backups-rg`)	~~Isolation~~Scaffolded, ~~boundary~~disabled — see §3

These are unrelated Azure storage accounts serving different data, set up at different times. Do not confuse them.

2. ACTIVE: LightRAG/Neo4j → `plockfrontstaging`

Script: ~/system/tools/lightrag-backup.sh LaunchAgent: com.alai.lightrag-backup.plist — weekly, Sunday 04:00 CEST Credentials: ~/system/config/azure-lightrag-backup.env (mode 0600) — AZURE_STORAGE_ACCOUNT, AZURE_STORAGE_KEY, AZURE_STORAGE_CONTAINER

What happens:

SSH to Azure VM vm-alai-lightrag (20.240.61.67)

docker compose stop lightrag neo4j (graceful, ~30-90s downtime)

Tar each of 4 Docker volumes (lightrag-data, lightrag-kg, lightrag-cache, lightrag-neo4j-data)

docker compose start neo4j lightrag

SHA-256 manifest + README written alongside the snapshot

SCP snapshot back to Mac Studio (~/system/backups/lightrag/, local safety net, keep last 4)

Azure Blob upload — az storage blob upload per file, --tier Cool, blob path <container>/<timestamp>/<file>

Azure-side rotation — keep last 8 snapshots (~8 weeks), older ones deleted via az storage blob delete-batch

Azure resource details:

Storage account: plockfrontstaging (swedencentral, Hot-tier account — a pre-existing Plock account being reused for this, not a dedicated ALAI backup ~~storage~~account)

~~Storage~~

Container: ~~Account~~lightrag-backup

Resource group: plock-staging-rg

Per-blob access tier: Cool

Retention: last 8 snapshots offsite (vs. 4 kept locally)

Cost: ~$0.05–0.10/month for ~4 GB retained

Restore: full step-by-step (download-batch, checksum verify, volume restore to Azure VM or to Mac Studio) is documented in LightRAG Backup runbook — do not duplicate here, that page is the source of truth for restore procedure.

3. SCAFFOLDED / DISABLED: SQLite → `alaibackups0ebb`Blob

~~storage~~

This path exists in config and credentials but is intentionally not the live path as of 2026-07-28.

Storage account: alaibackups0ebb, resource group alai-backups-rg Containers (~~LRS,~~per ~~Standard~~original ~~tier)~~design):

~~(Storage Blob Data Contributor)~~

Container	Purpose
~~Container~~	`system-db-backups`	SQLite ~~databases~~database blobs (~~hivemind.db,~~litestream ~~mission-control.db,~~`type: etc.)abs` replicas)
~~Container~~	`system-git-bundles`	Git ~~repository~~bundle ~~bundles~~exports
~~Service~~ ~~Principal~~	`alai-backup-writerbitwarden-exports`	~~Scoped~~Vaultwarden/Bitwarden ~~write-only~~export ~~access~~snapshots

Credentials: ~/system/config/azure-backup.env (mode 600)

AZURE_BACKUP_SP_APPID = 1a0b3018-0c31-474b-918f-531b0a29a669 (SP name: alai-backup-writer) — same appid used as the general-purpose az-cli identity elsewhere in the system, not a narrowly-scoped backup-only SP

AZURE_BACKUP_SP_TENANT = 3454a03f-20b4-4bda-a116-2293c459aecd

AZURE_SUBSCRIPTION_ID = 5b0b4d9b-e677-464e-abf0-5170cbce3b8e

VAULT_EXPORTER_APPID / VAULT_EXPORTER_SECRET = still FILL_AFTER_SP_CREATION — the narrower-scoped SP intended for the bitwarden-exports container only was never created. Do not assume it exists.

LaunchAgent: com.alai.azure-db-backup.plist.disabled (file dated 2026-04-20) — filename literally carries .disabled, RunAtLoad=false, never renamed to an active .plist. Confirmed present but inert on disk 2026-07-28.

Litestream (~/system/config/litestream.yml): documents alai-backup-writer as the intended Azure auth for streaming SQLite replication, and originally shipped replicas of type: abs (Azure Blob Storage) pointed at system-db-backups. As of 2026-07-28 all 64 configured databases use type: file (local) replicas only — zero type: abs replicas remain.

Why it's off — CEO decision, MC #105462 (2026-07-13)

Litestream's per-sync-tick ListBlobs full-prefix enumeration against alaibackups0ebb (driven mainly by two hot telemetry DBs — flywheel ~113K blobs, mission-control ~102K blobs) drove egress up to 518 GB/day (~370 NOK/day). After sync-interval tuning and lifecycle-rule fixes proved insufficient to bring it under control, CEO decision (verbatim): "Prebaci lokalno! Nemamo placene korisnika kad to dodje ide backup na azure do tada save money gdje mozemo." — move everything to local replicas; Azure Blob SQLite backup returns once ALAI has paying customers. Full incident record: Azure litestream egress saga + backup policy — MC #105462.

Reactivation checklist (when triggered)

Re-verify RBAC on alai-backups-rg / alaibackups0ebb — the current az-cli SP identity returned AuthorizationFailed reading this account/RG as of 2026-07-28; access is not currently confirmed.

Re-enable the LaunchAgent: rename com.alai.azure-db-backup.plist.disabled → com.alai.azure-db-backup.plist, launchctl load.

Re-point at least the P0-financial tier of litestream.yml replicas back to type: abs.

Re-scope the account-wide Azure lifecycle rule fixed under #105490 (it lacked prefixMatch and archived L0 blobs for 35 dormant DBs during the original incident) before turning replication back on.

Service Principal4. Setup commands (as originally provisioned, for reference)

Creating the general-purpose backup SP (already done — this is what AZURE_BACKUP_SP_APPID above corresponds to):

# Create service principal
az ad sp create-for-rbac --name alai-backup-writer --skip-assignmentrole # Assign "Storage Blob Data ContributorContributor" to\
  SA--scopes /subscriptions/5b0b4d9b-e677-464e-abf0-5170cbce3b8e/resourceGroups/alai-backups-rg/providers/Microsoft.Storage/storageAccounts/alaibackups0ebb

Creating the narrower vault-export-only SP (~~not~~documented, ~~subscription)~~never ~~STORAGE_ID=$(~~executed — VAULT_EXPORTER_APPID still unfilled):

az storagead accountsp showcreate-for-rbac --name alaibackups0ebb --query id -o tsv)
az role assignment create \
  --assignee <service-principal-app-id> \alai-vault-exporter --role "Storage Blob Data Contributor" \
  --scopescopes "$STORAGE_ID"/subscriptions/5b0b4d9b-e677-464e-abf0-5170cbce3b8e/resourceGroups/alai-backups-rg/providers/Microsoft.Storage/storageAccounts/alaibackups0ebb/blobServices/default/containers/bitwarden-exports
#

~~Store~~

~~credentials~~ in

ALAI Backup Strategy — full 4-layer picture (Git, SQLite, LightRAG, DR)

LightRAG Backup (Azure-native + local safety net) — full restore procedure for the active Azure path

Azure litestream egress saga + backup policy — MC #105462 — why the SQLite→Azure path was turned off

Disaster Recovery Runbook

Document Owner: Skillforge Last Verified: 2026-07-28 — ~/Library/LaunchAgents/com.alai.azure-db-backup.plist.disabled (present, inert), ~/system/config/litestream.yml (64/64 DBs type: file), ~/system/config/azure-backup.env ~~cat~~and ~~> ~/system/config/~~azure-lightrag-backup.env <

Lifecycle Policy

~~Hot → Cool:~~ ~~30 days~~
~~Cool → Archive:~~ ~~90 days~~
~~Archive → Delete:~~ ~~365 days~~
~~Delete blobs:~~ ~~Last modified > 365 days~~

az storage account management-policy create \
  --account-name alaibackups0ebb \
  --policy @lifecycle-policy.json

~~lifecycle-policy.json:~~

{
  "rules": [
    {
      "enabled": true,
      "name": "archive-old-backups",
      "type": "Lifecycle",
      "definition": {
        "actions": {
          "baseBlob": {
            "tierToCool": {"daysAfterModificationGreaterThan": 30},
            "tierToArchive": {"daysAfterModificationGreaterThan": 90},
            "delete": {"daysAfterModificationGreaterThan": 365}
          }
        },
        "filters": {"blobTypes": ["blockBlob"]}
      }
    }
  ]
}

Backup Scripts

LightRAG to Azure Blob

#!/bin/bash
# ~/system/tools/migrate-lightrag-to-azure.sh

source ~/system/config/azure-backup.env
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
BACKUP_FILE="/tmp/lightrag-backup-$TIMESTAMP.tar.gz"

tar -czf "$BACKUP_FILE" ~/system/lightrag/
az storage blob upload \
  --account-name alaibackups0ebb \
  --container-name system-db-backups \
  --name "lightrag-$TIMESTAMP.tar.gz" \
  --file "$BACKUP_FILE" \
  --auth-mode login

rm "$BACKUP_FILE"

Ollama Models Export

#!/bin/bash
# ~/system/tools/ollama-models-export.sh --azure

source ~/system/config/azure-backup.env
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
EXPORT_DIR="/tmp/ollama-export-$TIMESTAMP"

mkdir -p "$EXPORT_DIR"
ollama list | tail -n +2 | awk '{print $1}' > "$EXPORT_DIR/model-list.txt"

whilecontents read -r model; do
  ollama show "$model" --modelfile > "$EXPORT_DIR/$model.modelfile"
done < "$EXPORT_DIR/model-list.txt"

tar -czf "$EXPORT_DIR.tar.gz" "$EXPORT_DIR"
az storage blob upload \
  --account-name alaibackups0ebb \
  --container-name system-db-backups \
  --name "ollama-models-$TIMESTAMP.tar.gz" \
  --file "$EXPORT_DIR.tar.gz"

rm -rf "$EXPORT_DIR" "$EXPORT_DIR.tar.gz"

Disaster Recovery Path

~~List available backups:~~

az storage blob list \
  --account-name alaibackups0ebb \
  --container-name system-db-backups \
  --output table

~~Download latest backup:~~

az storage blob download \
  --account-name alaibackups0ebb \
  --container-name system-db-backups \
  --name "lightrag-20260420-143000.tar.gz" \
  --file /tmp/restore-lightrag.tar.gz

~~Verify SHA-256 checksum:~~

shasum -a 256 /tmp/restore-lightrag.tar.gz

~~Restore to target system:~~

tar -xzf /tmp/restore-lightrag.tar.gz -C ~/system/

Monitoring

~~Cron:~~ ~~Hourly backup at :15 (15 * * * *)~~

~~Log:~~directly), ~/system/logs/azure-tools/lightrag-backup.logsh
~~Alert:~~(script ~~HiveMind~~read ~~alert~~directly), ifcross-checked ~~backup~~against ~~fails 2 consecutive runs~~

node ~/system/agents/hivemind/hivemind.jsdocs/runbooks/alai-backup-strategy.md ~~post~~(same-day ~~john~~verification).

~~alert \ "Azure backup failed 2 consecutive runs — check ~/system/logs/azure-backup.log"~~

Azure Blob Offsite Backup Setup

Azure Blob Offsite Backup Setup

Overview

1. There are two independent Azure Blob backup paths — geographicone separationactive, fromone primary Sweden Central VM Retention: 365 days with lifecycle policies (Hot → Cool → Archive → Delete) Recovery Time Objective: 4 hours (manual restore)

Azure Resourcesscaffolded

2. ACTIVE: LightRAG/Neo4j → plockfrontstaging

3. SCAFFOLDED / DISABLED: SQLite → alaibackups0ebbBlob

Why it's off — CEO decision, MC #105462 (2026-07-13)

Reactivation checklist (when triggered)

Service Principal4. Setup commands (as originally provisioned, for reference)

Related

Lifecycle Policy

Backup Scripts

LightRAG to Azure Blob

Ollama Models Export

Disaster Recovery Path

Monitoring

1. There are two independent Azure Blob backup paths — geographicone separationactive, fromone primary Sweden Central VM
Retention: 365 days with lifecycle policies (Hot → Cool → Archive → Delete)
Recovery Time Objective: 4 hours (manual restore)

2. ACTIVE: LightRAG/Neo4j → `plockfrontstaging`

3. SCAFFOLDED / DISABLED: SQLite → `alaibackups0ebb`Blob