T5 — Kelsey DevOps fleet audit
DevOps + Observability + Fleet Health Audit (MC #10357 T5)
Auditor: Kelsey Hightower (T5 — DevOps, Platform Engineering) Date: 2026-04-30T22:02Z Method: Read-only. All data pulled from disk: `launchctl list`, `daemon-fleet-status.json`, log files, plist configs, cost-tracker.js, azure-db-backup.log, dr-sync.conf, litestream.yml. Scope: Full ALAI running fleet — 138 LaunchAgents, all backup systems, cost observability, DR posture.---
Q1 — Fleet Census & Health
Hard Numbers
| Category | Count | Source | |----------|-------|--------| | Total plists registered | 138 | `ls ~/Library/LaunchAgents/com.{john,alai}.*.plist \| wc -l` (94+44) | | Loaded in launchctl | 137 | `launchctl list \| grep -E "com.john.\|com.alai." \| wc -l` | | Running with PIDs | 47 | `launchctl list` — non-dash PID entries | | Calendar OK (scheduled, idle) | 69 | `daemon-fleet-status.json` | | Down exit_0 (conditional, trigger-gated) | 17 | `daemon-fleet-status.json` | | Error exits (non-zero last exit) | 4 | `daemon-fleet-status.json` | | Not loaded in launchd session | 2 | `daemon-fleet-status.json` + `launchctl` cross-check |
Fleet status JSON generated: `2026-04-30T19:50:00Z` File: `/Users/makinja/system/state/daemon-fleet-status.json` — 139 lines, 138 daemons, summary at EOF.
The 4 Hard Error Daemons (state = `calendar_err_256`)
These have `last_exit: 256` (bash exit 1 mapped through launchd). Every scheduled invocation fails.
1. `com.alai.azure-db-backup` — exit 256LastExitStatus = 256
Script: ~/system/daemons/azure-db-backup.sh
Log: ~/system/logs/azure-db-backup.out
LastExitStatus = 256
Mode: calendar
LastExitStatus = 256
Mode: calendar
LastExitStatus = 256
Log: ~/system/logs/infra-drift-detector.out
Last error (2026-04-30 05:51:02):
"warning: exhaustive rename detection was skipped due to too many files."
"diff.renameLimit variable to at least 3178"
"commit failed"
Additional Problem Daemons (not in the 4 hard errors, but flagged)
`com.john.legal-docs-azure-sync` — state: `down_exit_512`LastExitStatus = 512 (bash exit 2)
Log: ~/system/logs/legal-docs-azure-sync.log
Last errors (2026-04-30 03:47):
"ERROR: BlobAlreadyExists" — 84 of 84 files not uploaded
"source must be an existing directory" — ISBD sync path invalid
public-lumiscare: "fail 2500/2" — lumiscare.no unreachable (curl returns 000, DNS/CF issue)
public-docs: "fail 10/2" — docs.basicconsulting.no flagged down
public-vault: "fail 10/2" — vault.basicconsulting.no flagged down
[drain] CF credentials loaded from Vaultwarden
[drain] CF credentials refreshed
MC #10173 Status
MC #10173 ("Triage 11 silent daemon failures") was marked `done` at `2026-04-30T03:55:35`. However, the fleet watchdog snapshot at 19:50Z still shows `calendar_err_256` for `auto-verify-regression` and `infra-drift-detector`. Either the post-fix runs haven't been triggered, or the fixes were not applied. The original 11 failures from the memo:
| Daemon | Memo Status | Current State (19:50Z) | |--------|-------------|------------------------| | com.alai.azure-db-backup | exit 256 | STILL exit 256 (partial — see above) | | com.alai.bitwarden-vault-export | exit 256 | GONE — plist removed (good) | | com.alai.cert-expiry-monitor | exit 256 | `calendar_ok` exit 0 — FIXED | | com.john.b2-offsite-backup | exit 256 | `calendar_ok` exit 0 — FIXED | | com.john.lightrag-monitor | exit 512 | `calendar_ok` exit 0 — FIXED | | com.john.autowork | exit 256 | `calendar_ok` exit 0 — FIXED | | com.alai.aider-runner | KeepAlive exit 256 | NOT IN FLEET — plist removed | | com.alai.apply-knowledge | exit 256 | STILL exit 256 — UNFIXED | | com.john.auto-verify-regression | exit 256 | STILL exit 256 — UNFIXED | | com.john.infra-drift-detector | exit 256 | STILL exit 256 — UNFIXED | | com.john.legal-docs-azure-sync | KeepAlive exit 32512 | down_exit_512 — STILL FAILING (different error) |
Score: 5 fixed, 2 removed/gone, 4 still failing. MC #10173 should not have been closed with 4 open failures.---
Q2 — Observability Gap: Daily CEO-Facing Telemetry
Current State
`com.alai.john-daily-digest` runs at 08:00 daily and sends to Slack `#exec`. Last confirmed run: 2026-04-30T06:00Z. It includes:
- MC stats (open/done counts)
- Cost summary (today's total, tokens, failures)
- H-priority task count
- Signals (email, triage items)
`com.john.ops-watchdog` runs continuous health checks every 2 minutes and logs JSON to `~/system/logs/ops-watchdog.log`. No CEO-facing surface.
`com.alai.daemon-fleet-watchdog` runs every 15 minutes. No CEO-facing surface.
The Single Fix That Closes This Forever
A 09:00 daily Slack push to `#ceo-ops` (or `#exec`) combining fleet health + cost + key alerts.The existing pieces are all there:
Tool: node ~/system/tools/slack.js send <channel> "<message>"
Tool: node ~/system/tools/cost-tracker.js summary today
Tool: node ~/system/tools/mc.js stats
State: ~/system/state/daemon-fleet-status.json
Log: ~/system/logs/ops-watchdog.log (last cycle)
Content per run (8 lines in Slack):
ALAI Fleet Brief — 2026-04-30 09:00
Fleet: 46 running / 138 total | Errors: 4 | New failures: 0
Cost yesterday: $38,185 | This week: $368,509 | Trend: +2%
MC: 890 open | 1 in-progress | 29 awaiting review
Services: alai.no OK | docs OK | vault OK | lumiscare DOWN
Backups: azure-db OK (07:00) | litestream running | dr-sync OK (21:55)
Alerts: legal-docs-azure-sync FAILING (84 blobs BlobAlreadyExists)
pi-orch: IDLE (no eligible tasks since 19:58)
---
Q3 — Cost Observability
Today's Spend (2026-04-30)
Total requests: 243
Total cost: $129,209.12
Input tokens: 1,578,295,018
Output tokens: 182,750,690
Failures: 0
By model: claude-opus-4-7 176 req $128,546.95 claude-sonnet-4-6 67 req $662.17
This Week (2026-04-24 to 2026-04-30)
Total requests: 1,429
Total cost: $368,509.48
Input tokens: 5,018,933,916
Output tokens: 637,344,132
Failures: 12
By model: claude-opus-4-7 1,296 req $367,825.13 claude-sonnet-4-6 90 req $684.25 claude-haiku-4-5 1 req $0.11 ollama/qwen+llama 30 req $0.00
Is Cost Surfaced to CEO Automatically?
Partially. `john-daily-digest` (08:00) includes a one-line cost summary: `"Cost: $X, N in / N out tokens, N failures"`. This is the only CEO-facing cost signal.`cost-daily-report` writes `~/system/reports/cost-daily.md` at 23:55 but does not send it anywhere. The file is accurate but invisible unless actively queried.
What must be in a daily cost digest: 1. Today's spend vs yesterday (absolute + delta) 2. Week-to-date vs prior week run rate 3. Top 3 cost-generating tasks by MC ID (cost-tracker has `task-summary`) 4. Model distribution — is Opus being used where Sonnet/Haiku would do? 5. Any single run exceeding $1,000 (the Mehanik threshold) 6. Unbounded loop flag: any agent with >20 consecutive runs in 24hUnbounded Loop Risk: pi-orchestrator
The pi-orchestrator (PID 47730) is in IDLE state — cycling every 30 seconds, logging `"No eligible tasks"`. This is correct current behavior. However, the known prior issue (YouTube RAG during IDLE) is the `com.john.youtube-nightly-learning` daemon (PID 83439), which is a keepalive running `youtube-continuous-learning.sh`. At time of audit, it had 5 subprocesses (4 bash + 1 node), CPU at 0.0% total, processing "Best API Documentation Tools in 2024" (Ollama llama3.1:8b, not Claude). Not burning Claude tokens. Risk: if it escalates to Claude backends, each transcript analysis could be expensive. Current behavior is fine; it uses Ollama.
The $129K single-day figure and $368K weekly figure are the real concern. 176 Opus invocations today at an average of ~$730 each. With zero revenue, this burn rate requires CEO awareness daily, not just as a buried line in the morning digest.
---
Q4 — Backup & Disaster Recovery
mission-control.db
Litestream (PID 51452, keepalive, running) streams `mission-control.db` to Azure Blob Storage (`alaibackups0ebb.blob.core.windows.net/system-db-backups/litestream/mission-control`) with 1-second sync interval, 7-day retention. This is continuous WAL replication — not a snapshot. RTO: minutes (restore from latest WAL point). Last confirmed operational: litestream is running and keepalive PID is stable.Litestream covers 50+ databases total (P0 through P2 tiers). Full list in `/Users/makinja/system/config/litestream.yml`.
hivemind.db (knowledge graph)
Yes, backed up. Two mechanisms: 1. Litestream streaming replication (1s sync, 7d retention) — same ABS container 2. Qdrant snapshot via `azure-db-backup.sh`: the `hivemind` Qdrant collection is snapshotted and uploaded as `qdrant/2026-04-30/hivemind-2026-04-30.tar` (466M, SHA-256 verified at 09:50 today)Azure VM (BookStack / Vaultwarden / Documenso / Grafana / Planka)
No local backup mechanism found on ANVIL for the VM's own data. The `azure-db-backup.sh` backs up Docker volumes on ANVIL — not the Azure VM at 4.223.110.181.The Azure VM runs BookStack (wiki), Vaultwarden (password vault), Documenso (signing), Grafana, and Planka. None of these appear in `litestream.yml` (which points to `~/system/databases/` on ANVIL). No dr-sync target for 4.223.110.181 found in `dr-sync.conf` or `dr-sync.sh`.
If the Azure VM is deleted tonight, BookStack and Vaultwarden data is lost unless Azure itself has backup snapshots configured at the VM level. This is a significant gap.Confirmed: `docs.basicconsulting.no` (BookStack) returns HTTP 302 — service is live. `vault.basicconsulting.no` returns HTTP 302 — service is live. Both reachable as of 22:02Z today.
DR Sync (Mac Studio → Mac Mini/FORGE)
`dr-sync.sh` syncs to `10.0.0.2` (FORGE, Thunderbolt Bridge) every 6 hours. Last run: `2026-04-30T21:55:01` — 9/9 targets synced, 0 failures, 29 seconds. This covers `~/system/` on ANVIL.
One bug: the script calls `node ~/system/tools/slack.js send ops "..."` but the Slack channel `#ops` does not exist, producing `ERROR: Channel #ops not found` at end of every run. The sync itself succeeds; only the notification fails.
SSH Keys and Vault Credentials
`~/.ssh/azure_alai` (Azure VM key) and Bitwarden vault credentials exist on ANVIL. The `dr-sync.sh` syncs `~/system/` — but SSH keys live in `~/.ssh/`, which is in the protected zone per CLAUDE.md. Whether `~/.ssh/` is included in the Mac Mini DR sync is not confirmed from config alone.
`com.john.vault-keeper` (PID 87005) and `com.john.vault-proxy` (PID 1222) are both running — Bitwarden vault is accessible. The Vaultwarden instance on Azure VM is the credential store for many system secrets (CF Access, API keys). If the VM goes down, vault becomes inaccessible unless Bitwarden CLI can reach the VM.
If Laptop (ANVIL) Dies Tonight — What Is Lost?
| Asset | Status | |-------|--------| | mission-control.db | Safe — Litestream streaming to ABS | | hivemind.db | Safe — Litestream + Qdrant snapshots | | costs.db, events.db, ~48 other SQLite DBs | Safe — Litestream | | ~/system/ directory | Safe — dr-sync to FORGE (21:55 today) | | Qdrant collections (sessions, hivemind, knowledge) | Safe — daily snapshot to ABS | | LightRAG Docker volume (432M) | Safe — azure-db-backup runs 1-2x/day full | | Drop Postgres (prod) | Safe — azure-db-backup daily | | BookStack/Vaultwarden data on Azure VM | UNKNOWN — no local backup seen | | SSH keys (~/.ssh/) | UNCLEAR — not confirmed in dr-sync targets | | Installed binaries, Homebrew | LOST — not backed up, but reconstructible | | Local uncommitted work in git repos | LOST |
Conclusion: Core operational data is well-protected. The two gaps are the Azure VM data and SSH key portability.---
Q5 — Daily 5-Minute Fleet Health Ritual
Daily Ritual (5 minutes)
Run these commands in order each morning after boot:
1. Fleet error countnode ~/system/tools/mc.js stats
python3 -c "
import json; d=json.load(open('/Users/makinja/system/state/daemon-fleet-status.json'))['daemons']
errs=[k for k,v in d.items() if 'err' in v['state'] or 'exit_512' in v['state']]
print('ERRORS:', len(errs)); [print(' ', e) for e in errs]
"
node ~/system/tools/cost-tracker.js summary today
tail -3 ~/system/logs/azure-db-backup.out
tail -3 ~/system/logs/dr-sync.log
curl -sI https://docs.basicconsulting.no | head -1
curl -sI https://vault.basicconsulting.no | head -1
launchctl list com.alai.litestream | grep PID
tail -2 ~/system/logs/pi-orchestrator.log
---
Weekly Checklist (15 minutes, every Monday)
- Cost week-over-week: `node ~/system/tools/cost-tracker.js summary week` — compare to prior week. Flag any >20% increase.
- Full fleet error review: Read `~/system/state/daemon-fleet-status.json` summary block. All err states should be trending down over time, not accumulating.
- azure-db-backup stall audit: `grep "COMPLETE\|FAIL\|stall" ~/system/logs/azure-db-backup.out | tail -20` — count stalled runs vs complete runs. More than 2 stalls per day = timeout needs fixing.
- Archive-first compliance: `cat /tmp/archive-first-scan-report-$(date +%Y%m%d).txt` — should show decreasing candidate count as docs get archived. Current: 55 candidates with 0 ledger entries = 0 archiving done.
- ops-watchdog false positives: `grep "Service down" ~/system/logs/ops-watchdog.log | tail -20` — confirm public-docs and public-vault are still false-positives (302 treated as down). If lumiscare still 000, escalate.
- Litestream replication lag: `cat ~/system/logs/litestream.log | grep -i "error\|lag\|behind" | tail -10` — should be empty.
Monthly Checklist (30 minutes, first of month)
- Azure Blob inventory: Verify ABS container has current-month blobs for all litestream paths. Query: `az storage blob list --container-name system-db-backups --prefix "litestream/mission-control" --query "[].{name:name}" -o table` — confirm entries from current month.
- DR recovery drill: Restore `mission-control.db` from litestream ABS to `/tmp/mc-restore-test.db`. Verify row count matches live DB. `litestream restore -config ~/system/config/litestream.yml -replica mc-abs /tmp/mc-restore-test.db`
- Cost model distribution audit: `node ~/system/tools/cost-tracker.js task-summary` — identify top 5 cost-generating tasks. Are they justified? Is Opus being used where Sonnet is sufficient?
- Azure VM backup verification: SSH to `[email protected]` and verify BookStack/Vaultwarden data is intact. Until a VM-level backup policy is confirmed, this is the only assurance check for that data.
- SSH key rotation check: Verify `~/.ssh/azure_alai` key is documented in 1Password/Bitwarden and FORGE has a copy.
- Orphaned /tmp cleanup: `ls /tmp/alai-azure-backup- | wc -l` — currently 32 orphaned dirs from failed backup runs. These accumulate indefinitely. Add `rm -rf /tmp/alai-azure-backup-` to a weekly cron or the backup script's cleanup phase.
Top 5 Things Broken Right Now
1. CRITICAL — legal-docs-azure-sync has never worked Exit 512. Azure `BlobAlreadyExists` error — script lacks `--overwrite` flag. 84/84 legal documents (corporate, signed, insurance) have never been uploaded to Azure cold storage. The script also references a non-existent ISBD directory. ZAKON ARCHIVE FIRST is being violated for ALAI's most important documents. Fix: add `--overwrite` to `az storage blob upload-batch` call. File: `/Users/makinja/system/daemons/legal-docs-azure-sync.sh` 2. HIGH — Azure VM has no ANVIL-side backup BookStack (wiki), Vaultwarden (password vault), Documenso, Grafana, and Planka run on `4.223.110.181`. No backup mechanism targeting this VM was found in `litestream.yml`, `azure-db-backup.sh`, or `dr-sync.sh`. If the Azure VM is deleted or corrupted, operational knowledge base and all vault passwords are gone. Fix: add VM-level Azure Backup policy, or SSH-pull volumes nightly from ANVIL. 3. HIGH — $129K/day cost is invisible to CEO until morning digest The `cost-daily-report` writes to `~/system/reports/cost-daily.md` at 23:55 but never sends to Slack. The morning digest includes a one-line cost mention. If a runaway Opus loop fires at midnight, it won't be seen until 08:00 the next day — 8+ hours of potential unchecked spend. Fix: add a `cost-spike-alert.sh` triggered when any hour exceeds a threshold (e.g., $10K/hour), piped to `slack.js send exec`. 4. MEDIUM — infra-drift-detector fails every run (git rename limit) Every scheduled run fails because the underlying git repo has 3178+ renamed files and hits `diff.renameLimit`. The fix is a one-liner: `git config diff.renameLimit 0` in the script before the commit, or setting `SAFE_COMMIT=false` when only Brewfile drift is detected. Until fixed, drift detection is completely non-functional — the fleet could accumulate unauthorized plist/Brewfile changes without detection. File: `/Users/makinja/system/logs/infra-drift-detector.out` — error confirmed. 5. MEDIUM — ops-watchdog false alarms are drowning real signals `public-docs` and `public-vault` have been showing "Service down" every 2 minutes for at least 10 consecutive cycles (counter shows `fail 10/2`) when both services are actually live (HTTP 302). The watchdog script does not follow redirects. This is the boy-who-cried-wolf problem — when a real outage happens, it is invisible in a stream of false positives. Fix: add `--location` or `--max-redirs 1` to the curl check, or accept 3xx as healthy. Also: `lumiscare.no` shows `fail 2500/2` — that is a genuine outage, not false-positive.---
Bonus Observations
Slack `#ops` channel missing: `dr-sync.sh` calls `node ~/system/tools/slack.js send ops "..."` at the end of every run. The channel `#ops` does not exist, generating `ERROR: Channel #ops not found` in every dr-sync log. The sync itself is fine; only the notification is silently dropped. Fix: change to `#exec` or create `#ops`. lightrag-auto-heal not bootstrapped: Plist exists but not loaded in launchd session. The daemon designed to auto-recover LightRAG cannot run. If LightRAG goes down, the watchdog that would heal it is itself not watching. archive-first-ledger.jsonl is empty: `/Users/makinja/system/state/archive-first-ledger.jsonl` has 0 bytes / 0 lines. The scan correctly identifies 55 unarchived candidates (including signed corporate PDFs 50-76 days old). Zero have been archived. ZAKON ARCHIVE FIRST has been detecting violations since 2026-04-29 but no archiving action has been taken. 32 orphaned azure backup temp dirs in /tmp: `/tmp/alai-azure-backup-20260422` through `20260430` — stalled partial backup runs accumulate and are never cleaned. Each is ~700MB (lightrag + neo4j volumes). Total: potentially 20+ GB of temp data. Add `trap cleanup EXIT` to the backup script. Context usage JSONL is 34MB and growing: `/Users/makinja/system/logs/context-usage.jsonl` — 34.4MB at last modification 22:02Z. This file likely grows unbounded. Check if `com.john.log-rotate` covers it.---
Report generated by T5 Kelsey Hightower for Petter Graff (Chief Architect, T1 Synthesis). All findings are evidence-based — no memory assertions.
No comments to display
No comments to display