ANVIL DR Bootstrap Runbook (Mac Air)

ANVIL DR Bootstrap Runbook (Mac Air) 

 When to use 
 This runbook is for recovering the ALAI AI factory infrastructure when: 
 
 ANVIL (Mac Studio, 100.103.49.98) is dead, stolen, or inaccessible 
 Hardware failure requiring complete rebuild on new Mac 
 Setting up FORGE (disaster recovery clone) on fresh hardware 
 Provisioning a new MacBook Air for Alem with minimal AI factory capabilities 
 

 SPOF Context: As of 2026-04-20, ANVIL is the single Mac Studio hosting 112 LaunchAgent daemons, 68 SQLite databases (litestream-replicated), Ollama (8 models), and the entire ~/system + ~/.claude infrastructure. This runbook enables recovery to any fresh Mac with admin access. 

 

 Prerequisites 
 Before starting bootstrap, ensure you have: 
 
 Fresh Mac with admin account (macOS Sonoma or later, Apple Silicon preferred) 
 Tailscale app installed + logged into alembasic@ tailnet (download from tailscale.com/download) 
 GitHub account with read access to:
 
 github.com/johnatbasicas/clawd (~/system repo, auto-backup branch) 
 github.com/johnatbasicas/claude-config (~/.claude repo) 
 
 
 Bitwarden account unlocked with master password ready (Alem's personal vault: alembasic@gmail.com) 
 Internet connection (stable, for 2-3 GB of Homebrew packages + Ollama models) 
 

 

 Step-by-step Bootstrap 

 Phase 1: Foundation 

 1. Install Xcode Command Line Tools 
 xcode-select --install 
 Expected: GUI dialog appears. Click "Install" and wait 5-10 minutes. Verify with: 
 xcode-select -p
# Should output: /Library/Developer/CommandLineTools 

 2. Install Homebrew 
 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" 
 Expected: Homebrew installs to /opt/homebrew . Add to shell profile: 
 echo 'eval "$(/opt/homebrew/bin/brew shellenv)"' >> ~/.zprofile
eval "$(/opt/homebrew/bin/brew shellenv)"

# Verify:
brew --version
# Should show: Homebrew 4.x.x 

 3. Install Bitwarden CLI + unlock vault 
 brew install bitwarden-cli

# Unlock vault (enter master password when prompted):
bw login alembasic@gmail.com
export BW_SESSION=$(bw unlock --raw)

# Verify:
bw status | jq .status
# Should show: "unlocked" 
 Note: Keep this terminal window open. BW_SESSION is needed for bootstrap script. 

 

 Phase 2: Clone Infrastructure Repos 

 4. Clone ~/system (clawd repo) 
 # If using SSH (recommended if SSH keys already set up):
git clone git@github.com:johnatbasicas/clawd.git ~/system

# OR if using HTTPS with GitHub PAT:
git clone https://github.com/johnatbasicas/clawd.git ~/system

# Switch to auto-backup branch (contains latest portability artifacts):
cd ~/system
git checkout auto-backup
git pull 

 Expected: 
 ls ~/system/
# Should show: Brewfile, bootstrap.sh, config/, databases/, tools/, etc. 

 5. Clone ~/.claude (claude-config repo) 
 git clone git@github.com:johnatbasicas/claude-config.git ~/.claude

# Verify:
ls ~/.claude/
# Should show: CLAUDE.md, hooks/, agents/, skills/, projects/ 

 

 Phase 3: Run Bootstrap Script 

 6. Execute bootstrap (with BW_SESSION active) 
 cd ~/system
bash bootstrap.sh workstation 

 Role options: 
 
 anvil : Full primary node (all daemons, Ollama, heavy workloads) 
 forge : DR clone (continuous restore from Azure, lighter load) 
 workstation : Minimal setup (SSH relay to ANVIL for heavy ops) 
 

 What the script does: 
 
 Re-checks Xcode CLT + Homebrew (idempotent) 
 Installs ~70 brew packages from Brewfile (15-30 min depending on connection) 
 Copies 112 LaunchAgent plists from ~/system/config/launchagents/ to ~/Library/LaunchAgents/ 
 Rehydrates BW:<item> placeholders in plists by calling bw get password <item> 
 Loads all LaunchAgents via launchctl bootstrap 
 Verifies core services (Ollama, litestream) 
 

 Expected output (tail of bootstrap.log): 
 [bootstrap] Bootstrap COMPLETE. Next steps:
[bootstrap] - Verify SSH: ssh makinja@100.103.49.98
[bootstrap] - Check MC: node ~/system/tools/mc.js list
[bootstrap] - Log: /Users/makinja/bootstrap.log

LaunchAgents loaded: 112
Ollama models available: 8
Litestream: RUNNING 

 If BW rehydration fails: You'll see warnings like: 
 WARN: Bitwarden item 'groq-api-key' not found — com.alai.groq-model-benchmark.plist will need manual fix 
 Fix manually after bootstrap completes (see Troubleshooting section). 

 

 Phase 4: Database Restore (if DBs lost/corrupt) 

 When to run: Only if ~/system/databases/ is empty or you need to restore from Azure backups (e.g., ANVIL disk died). 

 7. Set Azure auth environment variables 
 export AZURE_CLIENT_ID="1a0b3018-0c31-474b-918f-531b0a29a669"
export AZURE_CLIENT_SECRET=$(bw get password alai-backup-writer-secret)
export AZURE_TENANT_ID="cd0a7929-1d14-4f81-820d-b36e45f72cf7" 

 8. Restore P0 critical databases 
 mkdir -p ~/system/databases

# Mission Control:
litestream restore -config ~/system/config/litestream.yml -if-replica-exists ~/system/databases/mission-control.db

# HiveMind:
litestream restore -config ~/system/config/litestream.yml -if-replica-exists ~/system/databases/hivemind.db

# Tasks:
litestream restore -config ~/system/config/litestream.yml -if-replica-exists ~/system/databases/tasks.db

# Costs:
litestream restore -config ~/system/config/litestream.yml -if-replica-exists ~/system/databases/costs.db

# Events:
litestream restore -config ~/system/config/litestream.yml -if-replica-exists ~/system/databases/events.db 

 9. Restore P0 financial databases 
 # Fiken (accounting cache):
litestream restore -config ~/system/config/litestream.yml -if-replica-exists ~/system/databases/fiken.db

# Invoices:
litestream restore -config ~/system/config/litestream.yml -if-replica-exists ~/system/databases/invoices.db

# Contracts:
litestream restore -config ~/system/config/litestream.yml -if-replica-exists ~/system/databases/contracts.db

# Leads:
litestream restore -config ~/system/config/litestream.yml -if-replica-exists ~/system/databases/leads.db 

 Note: The -if-replica-exists flag prevents overwriting local DB if it's newer than Azure backup. Omit this flag to force restore. 

 Bulk restore all 68 DBs (if needed): 
 for db in mission-control hivemind tasks costs events fiken invoices contracts leads \
 orchestrator-queue orchestrator-workers durable-runner session-index knowledge \
 emails email-inbox alem-directives agent-routing bee-index companies contacts \
 deploy-registry design-reviews distill documents drafts drift email-audit \
 email-briefing email-index email-tracking escalations facts flywheel goals \
 guardrails-audit health-events hivemind-archive master-control mc minions \
 observability orchestrator-events pipeline projects routing-outcomes skill-improvements \
 skill-registry sprint-pipeline strategy-tracker teams tenders tickets tool-audit \
 tool-registry trace-events applications-tracker baikal-caldav prompt-cache \
 prompt-metrics semantic-reuse-index stbs telemetry token-cost usage vcr bih-tenders browser-tasks; do
 echo "Restoring $db..."
 litestream restore -config ~/system/config/litestream.yml -if-replica-exists ~/system/databases/$db.db || echo "WARN: $db restore failed or skipped"
done 

 Verify restores: 
 ls -lh ~/system/databases/*.db | wc -l
# Should show: 68 (or close, depending on which DBs had replicas)

# Check specific DB integrity:
sqlite3 ~/system/databases/mission-control.db "PRAGMA integrity_check;"
# Should output: ok 

 

 Bitwarden Items Required 

 The following Bitwarden vault items MUST exist in Alem's vault before running bootstrap. These are referenced as BW:<item> placeholders in LaunchAgent plists: 

 
 Item Name Used By Purpose 
 alai-backup-writer-secret litestream, Azure backups Azure SP client secret for Storage Blob write access 
 cf-access-client-secret BookStack sync, CF-protected APIs Cloudflare Access client secret for docs.basicconsulting.no 
 groq-api-key Groq model benchmark daemon Groq API key for LLM model testing 
 slack-app-token Slack integration Slack app-level token for socket mode 
 slack-bot-token Slack integration Slack bot user OAuth token (xoxb-...) 
 

 How to verify items exist: 
 bw get item alai-backup-writer-secret --session $BW_SESSION
bw get item cf-access-client-secret --session $BW_SESSION
bw get item groq-api-key --session $BW_SESSION
bw get item slack-app-token --session $BW_SESSION
bw get item slack-bot-token --session $BW_SESSION 

 If missing: Contact Alem or check Vaultwarden (https://vault.basicconsulting.no) for backup credentials. These secrets are also in ANVIL's Keychain if ANVIL is still accessible. 

 

 Post-Bootstrap Verification 

 10. Check LaunchAgents loaded 
 launchctl list | grep -E "com\.(alai|john)\." | wc -l
# Expected: ~110-112 (depending on role) 

 11. Verify Ollama running 
 curl -s http://localhost:11434/api/tags | jq -r '.models[].name'
# Expected (ANVIL): qwen2.5-coder:32b, llama3.3, deepseek-r1, etc. 

 12. Verify litestream replication 
 pgrep -fl litestream
# Should show: litestream replicate -config /Users/makinja/system/config/litestream.yml

# Check logs:
tail -f ~/system/logs/litestream.log
# Should show periodic sync messages (every 1-30s depending on DB tier) 

 13. Test Mission Control 
 node ~/system/tools/mc.js stats
# Should show task counts, agents, recent activity

node ~/system/tools/mc.js list --limit 5
# Should show recent tasks 

 14. Test SSH to original ANVIL (if still alive) 
 ssh makinja@100.103.49.98 "hostname && uptime"
# Expected: ANVIL + uptime if machine is reachable 

 

 Troubleshooting 

 Error: "brew: command not found" after install 
 Cause: Homebrew not in PATH. 
 Fix: 
 eval "$(/opt/homebrew/bin/brew shellenv)"
echo 'eval "$(/opt/homebrew/bin/brew shellenv)"' >> ~/.zprofile 

 Error: "bw: command not found" 
 Cause: Bitwarden CLI not installed or not in PATH. 
 Fix: 
 brew install bitwarden-cli
hash -r # Refresh shell PATH cache 

 LaunchAgent fails to load 
 Symptoms: launchctl bootstrap returns error code 119, 122, or 125. 
 Debug: 
 # Check specific agent status:
launchctl print gui/$(id -u)/com.alai.litestream
# Look for "state = waiting" or "last exit code"

# Check agent logs:
tail -f ~/system/logs/litestream.log
tail -f ~/Library/Logs/com.alai.*.log 

 Common exit codes: 
 
 119 : Invalid plist XML (malformed after sed replacement) 
 122 : Path not found (e.g., /Users/makinja hardcoded but new user is /Users/alem) 
 125 : Permission denied (env var secret not readable) 
 

 Secret rehydration failed 
 Symptoms: Bootstrap log shows "WARN: Bitwarden item 'X' not found". 
 Fix manually: 
 # Get secret from Bitwarden:
SECRET=$(bw get password groq-api-key --session $BW_SESSION)

# Edit plist:
vi ~/Library/LaunchAgents/com.alai.groq-model-benchmark.plist

# Replace BW:groq-api-key with actual value in <string> tag

# Reload:
launchctl bootout gui/$(id -u)/com.alai.groq-model-benchmark
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.alai.groq-model-benchmark.plist 

 Hardcoded /Users/makinja path mismatch 
 Cause: LaunchAgent plists contain hardcoded paths to /Users/makinja, but new Mac has different username (e.g., /Users/alem). 
 Fix (bulk replace): 
 NEW_USER=$(whoami)
cd ~/Library/LaunchAgents

for plist in com.alai.*.plist com.john.*.plist; do
 sed -i.bak "s|/Users/makinja|/Users/$NEW_USER|g" "$plist"
done

# Reload all:
launchctl bootout gui/$(id -u)
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/*.plist 

 Ollama models missing 
 Cause: Fresh install has no models cached. Models are NOT in git repos (too large). 
 Fix (pull from Ollama registry): 
 ollama pull qwen2.5-coder:32b
ollama pull llama3.3:70b
ollama pull deepseek-r1:32b
ollama pull deepseek-r1:70b
ollama pull devstral:24b
ollama pull mistral-small
ollama pull llama3.2-vision:90b
ollama pull qwq:32b

# Verify:
ollama list 

 Expected download size: ~150 GB total for all models. This takes 2-6 hours on good connection. 

 Database restore fails with "replica not found" 
 Cause: Azure credentials invalid, or DB was never replicated (new DB created after litestream setup). 
 Debug: 
 # Test Azure auth:
az login --service-principal \
 --username $AZURE_CLIENT_ID \
 --password $AZURE_CLIENT_SECRET \
 --tenant $AZURE_TENANT_ID

# List backups:
litestream snapshots -config ~/system/config/litestream.yml ~/system/databases/mission-control.db

# Should show timestamps of snapshots in Azure Blob Storage 

 If no snapshots: DB is new or replication was broken. Accept data loss or restore from other source (e.g., Time Machine if on ANVIL). 

 

 Known Limitations 

 
 Untested end-to-end: bootstrap.sh has NOT been tested on a completely fresh Mac. Code paths for Xcode install prompt, Homebrew first-run, and BW unlock flow are based on best practices but unverified in production DR scenario. 
 User rename not handled: If new Mac username != "makinja", LaunchAgent plists will fail due to hardcoded /Users/makinja paths. Manual sed replacement required (see Troubleshooting). 
 npm install layer incomplete: ~/system/tools/ contains 1,310 scripts, some requiring npm install in subdirs. bootstrap.sh does NOT auto-install these deps. Expect some tools to fail until deps are installed manually. 
 Ollama models not in backup: Models are fetched from Ollama registry on first use. Expect 2-6 hour delay to repopulate model cache (~150 GB). 
 GitHub auth assumed: Script assumes SSH keys or PAT for GitHub already configured. If not, git clone will prompt interactively. 
 No Keychain sync: macOS Keychain items (SSH keys, app passwords, etc.) are NOT part of this backup. Alem must re-enter credentials for Mail.app, Calendar, etc. 
 No ~/felles or ~/Documents: User data directories are NOT backed up by this system. Rely on Time Machine or iCloud for personal files. 
 

 

 Testing Recommendations 

 Before trusting this runbook in a real disaster: 
 
 Spin up a fresh Mac VM (UTM or Parallels) with macOS Sonoma 
 Run through Steps 1-6 end-to-end without looking at ANVIL 
 Verify LaunchAgent load count matches expected (~112) 
 Verify DB restore works for at least mission-control.db and hivemind.db 
 Document any new errors or missing secrets in this runbook 
 

 Assigned to: Petter Graff (CodeCraft) — MC task #8534 

 

 Last updated: 2026-04-20 | MC Task: #8534 | Tags: status=draft-untested, type=runbook, severity=critical