# ANVIL DR Bootstrap Runbook (Mac Air) # ANVIL DR Bootstrap Runbook (Mac Air) ## When to use This runbook is for recovering the ALAI AI factory infrastructure when: - ANVIL (Mac Studio, 100.103.49.98) is dead, stolen, or inaccessible - Hardware failure requiring complete rebuild on new Mac - Setting up FORGE (disaster recovery clone) on fresh hardware - Provisioning a new MacBook Air for Alem with minimal AI factory capabilities **SPOF Context:** As of 2026-04-20, ANVIL is the single Mac Studio hosting 112 LaunchAgent daemons, 68 SQLite databases (litestream-replicated), Ollama (8 models), and the entire ~/system + ~/.claude infrastructure. This runbook enables recovery to any fresh Mac with admin access. --- ## Prerequisites Before starting bootstrap, ensure you have: 1. **Fresh Mac** with admin account (macOS Sonoma or later, Apple Silicon preferred) 2. **Tailscale app** installed + logged into `alembasic@` tailnet (download from tailscale.com/download) 3. **GitHub account** with read access to: - `github.com/johnatbasicas/clawd` (~/system repo, auto-backup branch) - `github.com/johnatbasicas/claude-config` (~/.claude repo) 4. **Bitwarden account unlocked** with master password ready (Alem's personal vault: alembasic@gmail.com) 5. **Internet connection** (stable, for 2-3 GB of Homebrew packages + Ollama models) --- ## Step-by-step Bootstrap ### Phase 1: Foundation #### 1. Install Xcode Command Line Tools ``` xcode-select --install ``` **Expected:** GUI dialog appears. Click "Install" and wait 5-10 minutes. Verify with: ``` xcode-select -p # Should output: /Library/Developer/CommandLineTools ``` #### 2. Install Homebrew ``` /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" ``` **Expected:** Homebrew installs to `/opt/homebrew`. Add to shell profile: ``` echo 'eval "$(/opt/homebrew/bin/brew shellenv)"' >> ~/.zprofile eval "$(/opt/homebrew/bin/brew shellenv)" # Verify: brew --version # Should show: Homebrew 4.x.x ``` #### 3. Install Bitwarden CLI + unlock vault ``` brew install bitwarden-cli # Unlock vault (enter master password when prompted): bw login alembasic@gmail.com export BW_SESSION=$(bw unlock --raw) # Verify: bw status | jq .status # Should show: "unlocked" ``` **Note:** Keep this terminal window open. `BW_SESSION` is needed for bootstrap script. --- ### Phase 2: Clone Infrastructure Repos #### 4. Clone ~/system (clawd repo) ``` # If using SSH (recommended if SSH keys already set up): git clone git@github.com:johnatbasicas/clawd.git ~/system # OR if using HTTPS with GitHub PAT: git clone https://github.com/johnatbasicas/clawd.git ~/system # Switch to auto-backup branch (contains latest portability artifacts): cd ~/system git checkout auto-backup git pull ``` **Expected:** ``` ls ~/system/ # Should show: Brewfile, bootstrap.sh, config/, databases/, tools/, etc. ``` #### 5. Clone ~/.claude (claude-config repo) ``` git clone git@github.com:johnatbasicas/claude-config.git ~/.claude # Verify: ls ~/.claude/ # Should show: CLAUDE.md, hooks/, agents/, skills/, projects/ ``` --- ### Phase 3: Run Bootstrap Script #### 6. Execute bootstrap (with BW\_SESSION active) ``` cd ~/system bash bootstrap.sh workstation ``` **Role options:** - `anvil`: Full primary node (all daemons, Ollama, heavy workloads) - `forge`: DR clone (continuous restore from Azure, lighter load) - `workstation`: Minimal setup (SSH relay to ANVIL for heavy ops) **What the script does:** 1. Re-checks Xcode CLT + Homebrew (idempotent) 2. Installs ~70 brew packages from Brewfile (15-30 min depending on connection) 3. Copies 112 LaunchAgent plists from `~/system/config/launchagents/` to `~/Library/LaunchAgents/` 4. Rehydrates `BW:` placeholders in plists by calling `bw get password ` 5. Loads all LaunchAgents via `launchctl bootstrap` 6. Verifies core services (Ollama, litestream) **Expected output (tail of bootstrap.log):** ``` [bootstrap] Bootstrap COMPLETE. Next steps: [bootstrap] - Verify SSH: ssh makinja@100.103.49.98 [bootstrap] - Check MC: node ~/system/tools/mc.js list [bootstrap] - Log: /Users/makinja/bootstrap.log LaunchAgents loaded: 112 Ollama models available: 8 Litestream: RUNNING ``` **If BW rehydration fails:** You'll see warnings like: ``` WARN: Bitwarden item 'groq-api-key' not found — com.alai.groq-model-benchmark.plist will need manual fix ``` Fix manually after bootstrap completes (see Troubleshooting section). --- ### Phase 4: Database Restore (if DBs lost/corrupt) **When to run:** Only if ~/system/databases/ is empty or you need to restore from Azure backups (e.g., ANVIL disk died). #### 7. Set Azure auth environment variables ``` export AZURE_CLIENT_ID="1a0b3018-0c31-474b-918f-531b0a29a669" export AZURE_CLIENT_SECRET=$(bw get password alai-backup-writer-secret) export AZURE_TENANT_ID="cd0a7929-1d14-4f81-820d-b36e45f72cf7" ``` #### 8. Restore P0 critical databases ``` mkdir -p ~/system/databases # Mission Control: litestream restore -config ~/system/config/litestream.yml -if-replica-exists ~/system/databases/mission-control.db # HiveMind: litestream restore -config ~/system/config/litestream.yml -if-replica-exists ~/system/databases/hivemind.db # Tasks: litestream restore -config ~/system/config/litestream.yml -if-replica-exists ~/system/databases/tasks.db # Costs: litestream restore -config ~/system/config/litestream.yml -if-replica-exists ~/system/databases/costs.db # Events: litestream restore -config ~/system/config/litestream.yml -if-replica-exists ~/system/databases/events.db ``` #### 9. Restore P0 financial databases ``` # Fiken (accounting cache): litestream restore -config ~/system/config/litestream.yml -if-replica-exists ~/system/databases/fiken.db # Invoices: litestream restore -config ~/system/config/litestream.yml -if-replica-exists ~/system/databases/invoices.db # Contracts: litestream restore -config ~/system/config/litestream.yml -if-replica-exists ~/system/databases/contracts.db # Leads: litestream restore -config ~/system/config/litestream.yml -if-replica-exists ~/system/databases/leads.db ``` **Note:** The `-if-replica-exists` flag prevents overwriting local DB if it's newer than Azure backup. Omit this flag to force restore. **Bulk restore all 68 DBs (if needed):** ``` for db in mission-control hivemind tasks costs events fiken invoices contracts leads \ orchestrator-queue orchestrator-workers durable-runner session-index knowledge \ emails email-inbox alem-directives agent-routing bee-index companies contacts \ deploy-registry design-reviews distill documents drafts drift email-audit \ email-briefing email-index email-tracking escalations facts flywheel goals \ guardrails-audit health-events hivemind-archive master-control mc minions \ observability orchestrator-events pipeline projects routing-outcomes skill-improvements \ skill-registry sprint-pipeline strategy-tracker teams tenders tickets tool-audit \ tool-registry trace-events applications-tracker baikal-caldav prompt-cache \ prompt-metrics semantic-reuse-index stbs telemetry token-cost usage vcr bih-tenders browser-tasks; do echo "Restoring $db..." litestream restore -config ~/system/config/litestream.yml -if-replica-exists ~/system/databases/$db.db || echo "WARN: $db restore failed or skipped" done ``` **Verify restores:** ``` ls -lh ~/system/databases/*.db | wc -l # Should show: 68 (or close, depending on which DBs had replicas) # Check specific DB integrity: sqlite3 ~/system/databases/mission-control.db "PRAGMA integrity_check;" # Should output: ok ``` --- ## Bitwarden Items Required The following Bitwarden vault items MUST exist in Alem's vault before running bootstrap. These are referenced as `BW:` placeholders in LaunchAgent plists:
Item NameUsed ByPurpose
`alai-backup-writer-secret`litestream, Azure backupsAzure SP client secret for Storage Blob write access
`cf-access-client-secret`BookStack sync, CF-protected APIsCloudflare Access client secret for docs.basicconsulting.no
`groq-api-key`Groq model benchmark daemonGroq API key for LLM model testing
`slack-app-token`Slack integrationSlack app-level token for socket mode
`slack-bot-token`Slack integrationSlack bot user OAuth token (xoxb-...)
**How to verify items exist:** ``` bw get item alai-backup-writer-secret --session $BW_SESSION bw get item cf-access-client-secret --session $BW_SESSION bw get item groq-api-key --session $BW_SESSION bw get item slack-app-token --session $BW_SESSION bw get item slack-bot-token --session $BW_SESSION ``` **If missing:** Contact Alem or check Vaultwarden (https://vault.basicconsulting.no) for backup credentials. These secrets are also in ANVIL's Keychain if ANVIL is still accessible. --- ## Post-Bootstrap Verification #### 10. Check LaunchAgents loaded ``` launchctl list | grep -E "com\.(alai|john)\." | wc -l # Expected: ~110-112 (depending on role) ``` #### 11. Verify Ollama running ``` curl -s http://localhost:11434/api/tags | jq -r '.models[].name' # Expected (ANVIL): qwen2.5-coder:32b, llama3.3, deepseek-r1, etc. ``` #### 12. Verify litestream replication ``` pgrep -fl litestream # Should show: litestream replicate -config /Users/makinja/system/config/litestream.yml # Check logs: tail -f ~/system/logs/litestream.log # Should show periodic sync messages (every 1-30s depending on DB tier) ``` #### 13. Test Mission Control ``` node ~/system/tools/mc.js stats # Should show task counts, agents, recent activity node ~/system/tools/mc.js list --limit 5 # Should show recent tasks ``` #### 14. Test SSH to original ANVIL (if still alive) ``` ssh makinja@100.103.49.98 "hostname && uptime" # Expected: ANVIL + uptime if machine is reachable ``` --- ## Troubleshooting ### Error: "brew: command not found" after install **Cause:** Homebrew not in PATH. **Fix:** ``` eval "$(/opt/homebrew/bin/brew shellenv)" echo 'eval "$(/opt/homebrew/bin/brew shellenv)"' >> ~/.zprofile ``` ### Error: "bw: command not found" **Cause:** Bitwarden CLI not installed or not in PATH. **Fix:** ``` brew install bitwarden-cli hash -r # Refresh shell PATH cache ``` ### LaunchAgent fails to load **Symptoms:** `launchctl bootstrap` returns error code 119, 122, or 125. **Debug:** ``` # Check specific agent status: launchctl print gui/$(id -u)/com.alai.litestream # Look for "state = waiting" or "last exit code" # Check agent logs: tail -f ~/system/logs/litestream.log tail -f ~/Library/Logs/com.alai.*.log ``` **Common exit codes:** - `119`: Invalid plist XML (malformed after sed replacement) - `122`: Path not found (e.g., /Users/makinja hardcoded but new user is /Users/alem) - `125`: Permission denied (env var secret not readable) ### Secret rehydration failed **Symptoms:** Bootstrap log shows "WARN: Bitwarden item 'X' not found". **Fix manually:** ``` # Get secret from Bitwarden: SECRET=$(bw get password groq-api-key --session $BW_SESSION) # Edit plist: vi ~/Library/LaunchAgents/com.alai.groq-model-benchmark.plist # Replace BW:groq-api-key with actual value in tag # Reload: launchctl bootout gui/$(id -u)/com.alai.groq-model-benchmark launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.alai.groq-model-benchmark.plist ``` ### Hardcoded /Users/makinja path mismatch **Cause:** LaunchAgent plists contain hardcoded paths to /Users/makinja, but new Mac has different username (e.g., /Users/alem). **Fix (bulk replace):** ``` NEW_USER=$(whoami) cd ~/Library/LaunchAgents for plist in com.alai.*.plist com.john.*.plist; do sed -i.bak "s|/Users/makinja|/Users/$NEW_USER|g" "$plist" done # Reload all: launchctl bootout gui/$(id -u) launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/*.plist ``` ### Ollama models missing **Cause:** Fresh install has no models cached. Models are NOT in git repos (too large). **Fix (pull from Ollama registry):** ``` ollama pull qwen2.5-coder:32b ollama pull llama3.3:70b ollama pull deepseek-r1:32b ollama pull deepseek-r1:70b ollama pull devstral:24b ollama pull mistral-small ollama pull llama3.2-vision:90b ollama pull qwq:32b # Verify: ollama list ``` **Expected download size:** ~150 GB total for all models. This takes 2-6 hours on good connection. ### Database restore fails with "replica not found" **Cause:** Azure credentials invalid, or DB was never replicated (new DB created after litestream setup). **Debug:** ``` # Test Azure auth: az login --service-principal \ --username $AZURE_CLIENT_ID \ --password $AZURE_CLIENT_SECRET \ --tenant $AZURE_TENANT_ID # List backups: litestream snapshots -config ~/system/config/litestream.yml ~/system/databases/mission-control.db # Should show timestamps of snapshots in Azure Blob Storage ``` **If no snapshots:** DB is new or replication was broken. Accept data loss or restore from other source (e.g., Time Machine if on ANVIL). --- ## Known Limitations - **Untested end-to-end:** bootstrap.sh has NOT been tested on a completely fresh Mac. Code paths for Xcode install prompt, Homebrew first-run, and BW unlock flow are based on best practices but unverified in production DR scenario. - **User rename not handled:** If new Mac username != "makinja", LaunchAgent plists will fail due to hardcoded /Users/makinja paths. Manual sed replacement required (see Troubleshooting). - **npm install layer incomplete:** ~/system/tools/ contains 1,310 scripts, some requiring `npm install` in subdirs. bootstrap.sh does NOT auto-install these deps. Expect some tools to fail until deps are installed manually. - **Ollama models not in backup:** Models are fetched from Ollama registry on first use. Expect 2-6 hour delay to repopulate model cache (~150 GB). - **GitHub auth assumed:** Script assumes SSH keys or PAT for GitHub already configured. If not, git clone will prompt interactively. - **No Keychain sync:** macOS Keychain items (SSH keys, app passwords, etc.) are NOT part of this backup. Alem must re-enter credentials for Mail.app, Calendar, etc. - **No ~/felles or ~/Documents:** User data directories are NOT backed up by this system. Rely on Time Machine or iCloud for personal files. --- ## Testing Recommendations **Before trusting this runbook in a real disaster:** 1. Spin up a fresh Mac VM (UTM or Parallels) with macOS Sonoma 2. Run through Steps 1-6 end-to-end without looking at ANVIL 3. Verify LaunchAgent load count matches expected (~112) 4. Verify DB restore works for at least mission-control.db and hivemind.db 5. Document any new errors or missing secrets in this runbook **Assigned to:** Petter Graff (CodeCraft) — MC task #8534 --- *Last updated: 2026-04-20 | MC Task: #8534 | Tags: status=draft-untested, type=runbook, severity=critical*