Runbooks

Service runbooks — troubleshooting and recovery

BookStack Runbook

Runbook: BookStack

Service Type: Wiki / Knowledge Base Container: bookstack (lscr.io/linuxserver/bookstack:latest) Ports: 6875 (external) → 80 (internal) Internal URL: http://localhost:6875 External URL: http://192.168.68.61:6875 (LAN only, no Cloudflare tunnel yet) Database: MariaDB (bookstack_db) Compose File: ~/system/services/bookstack/docker-compose.yml

Service Info

BookStack is the documentation wiki for BasicAS Group. Stores runbooks, system docs, org info.

Stack:

Access:

API:

Status Check

Container Health

docker ps | grep bookstack

Expected output:

bookstack       Up X hours
bookstack_db    Up X hours

HTTP Check

curl -I http://localhost:6875

Expected: 200 OK or 302 Found

API Check

curl -s -H "Authorization: Token alai-v2-84c8e63775a52492:ff80c10c7c881d5dbf341500b3e826309a8570a11277d887143305d975b076de" http://localhost:6875/api/docs.json | head -5

Expected: JSON response with API docs.

Database Check

docker exec bookstack_db mariadb -u bookstack -p'8CdydCxVBD7wBoCVRXZE' bookstackapp -e "SELECT count(*) FROM pages;"

Restart Procedure

Quick Restart (Container Only)

docker restart bookstack

Full Stack Restart (Container + Database)

cd ~/system/services/bookstack
docker compose down
docker compose up -d

Wait 30 seconds, then verify:

docker ps | grep bookstack
curl -I http://localhost:6875

Sync System Docs to BookStack

BookStack is auto-populated from ~/system/ using the sync tool.

Sync All Mapped Content

node ~/system/tools/bookstack-sync.js sync

Sync Single File

node ~/system/tools/bookstack-sync.js sync ~/system/rules/development.md

Check Sync Status

node ~/system/tools/bookstack-sync.js status

Force Overwrite All

node ~/system/tools/bookstack-sync.js push

Mapping File: ~/system/config/bookstack-sync-map.json State File: ~/system/config/bookstack-sync-state.json

Troubleshooting

Problem: Container won't start

Check logs:

docker logs bookstack --tail 100

Common causes:

  1. Database not ready - wait 30s and retry
  2. Port 6875 already bound - check lsof -i :6875
  3. Volume permission issues - check ~/system/services/bookstack/data/

Fix:

cd ~/system/services/bookstack
docker compose down
docker compose up -d bookstack_db
sleep 30
docker compose up -d bookstack

Problem: Can't login (wrong password)

Check if admin credentials were changed in UI:

Reset admin password:

docker exec -it bookstack php /app/www/artisan bookstack:create-admin --email=admin@admin.com --name=Admin --password=newpassword

Problem: API returns 401 Unauthorized

Check token exists:

cat ~/system/config/bookstack.json

Regenerate token in UI:

  1. Login to BookStack
  2. Go to Settings → API Tokens
  3. Create new token
  4. Update ~/system/config/bookstack.json

Problem: Sync tool fails (500 error)

Check BookStack is running:

curl -I http://localhost:6875

Check API endpoint:

curl -s -H "Authorization: Token alai-v2-84c8e63775a52492:ff80c10c7c881d5dbf341500b3e826309a8570a11277d887143305d975b076de" http://localhost:6875/api/shelves | head -20

Check logs:

docker logs bookstack --tail 100

Problem: Database connection issues

Check database health:

docker exec bookstack_db mariadb-admin -u bookstack -p'8CdydCxVBD7wBoCVRXZE' ping

Expected: mysqld is alive

Check connection settings:

docker exec bookstack env | grep DB_

Expected:

DB_HOST=bookstack_db
DB_PORT=3306
DB_USERNAME=bookstack
DB_PASSWORD=8CdydCxVBD7wBoCVRXZE
DB_DATABASE=bookstackapp

API Usage

List Shelves

curl -s -H "Authorization: Token alai-v2-84c8e63775a52492:ff80c10c7c881d5dbf341500b3e826309a8570a11277d887143305d975b076de" http://localhost:6875/api/shelves

List Books

curl -s -H "Authorization: Token alai-v2-84c8e63775a52492:ff80c10c7c881d5dbf341500b3e826309a8570a11277d887143305d975b076de" http://localhost:6875/api/books

List Pages

curl -s -H "Authorization: Token alai-v2-84c8e63775a52492:ff80c10c7c881d5dbf341500b3e826309a8570a11277d887143305d975b076de" http://localhost:6875/api/pages

Create Page

curl -X POST -H "Authorization: Token alai-v2-84c8e63775a52492:ff80c10c7c881d5dbf341500b3e826309a8570a11277d887143305d975b076de" \
  -H "Content-Type: application/json" \
  -d '{"book_id":1,"name":"Page Title","markdown":"# Content"}' \
  http://localhost:6875/api/pages

Full API docs: http://localhost:6875/api/docs

Dependencies

Backup

Database Dump

docker exec bookstack_db mariadb-dump -u bookstack -p'8CdydCxVBD7wBoCVRXZE' bookstackapp | gzip > ~/backups/bookstack-$(date +%Y%m%d-%H%M%S).sql.gz

Data Volumes (includes uploads, images)

cd ~/system/services/bookstack
tar -czf ~/backups/bookstack-data-$(date +%Y%m%d-%H%M%S).tar.gz data/

Restore from Backup

# Stop service
cd ~/system/services/bookstack
docker compose down

# Restore database
gunzip -c ~/backups/bookstack-YYYYMMDD-HHMMSS.sql.gz | docker exec -i bookstack_db mariadb -u bookstack -p'8CdydCxVBD7wBoCVRXZE' bookstackapp

# Restore data (if needed)
cd ~/system/services/bookstack
tar -xzf ~/backups/bookstack-data-YYYYMMDD-HHMMSS.tar.gz

# Start service
docker compose up -d

Configuration

Key Environment Variables

Full config: ~/system/services/bookstack/docker-compose.yml

Application Settings (via UI)

Content Structure

BookStack organizes content as:

Shelf (top-level category)
  └─ Book (collection of pages)
       └─ Page (markdown document)
            └─ Chapter (optional grouping)

Current structure (as of 2026-02-10):

Notes

Last updated: 2026-02-10 Maintained by: John (AI Director)

Mattermost Runbook

Status: DEPRECATED 2026-05-18 — mm.basicconsulting.no decommissioned per CEO answer #5. Replace with comment # DEPRECATED 2026-05-18 or delete if easy.

Runbook: Mattermost

Service Type: Team Communication Platform Container: mattermost (mattermost/mattermost-team-edition:latest) Ports: 8065 (internal + external) External URL: https://mm.basicconsulting.no Database: PostgreSQL 15 (mattermost-db) Compose File: ~/system/services/mattermost/docker-compose.yml

Service Info

Mattermost is the primary team communication platform for BasicAS Group. Runs via Docker Compose with PostgreSQL backend.

Stack:

External Access:

Admin Access:

Status Check

Container Health

docker ps | grep mattermost

Expected output:

mattermost      Up X hours (healthy)
mattermost-db   Up X hours

HTTP Check

curl -I http://localhost:8065

Expected: 200 OK

External Access Check

curl -I https://mm.basicconsulting.no

Expected: 200 OK

Database Check

docker exec mattermost-db psql -U mmuser -d mattermost -c "SELECT count(*) FROM users;"

Restart Procedure

Quick Restart (Container Only)

docker restart mattermost

Full Stack Restart (Container + Database)

cd ~/system/services/mattermost
docker compose down
docker compose up -d

Wait 30-60 seconds for healthcheck to pass, then verify:

docker ps | grep mattermost
curl -I http://localhost:8065

Troubleshooting

Problem: Container won't start

Check logs:

docker logs mattermost --tail 100

Common causes:

  1. Database not ready - wait 30s and retry
  2. Port 8065 already bound - check lsof -i :8065
  3. Volume permission issues - check ~/system/services/mattermost/data/

Fix:

cd ~/system/services/mattermost
docker compose down
docker compose up -d mattermost-db
sleep 30
docker compose up -d mattermost

Problem: Login issues (can't sign in)

Check SMTP:

docker exec mattermost cat /mattermost/config/config.json | grep -A5 EmailSettings

Reset admin password:

docker exec -it mattermost mattermost user reset_password <user-email>

Problem: WebSocket errors (messages not real-time)

Check site URL:

docker exec mattermost env | grep MM_SERVICESETTINGS_SITEURL

Expected: MM_SERVICESETTINGS_SITEURL=https://mm.basicconsulting.no

If wrong, update in docker-compose.yml and restart.

Problem: Database connection issues

Check database health:

docker exec mattermost-db pg_isready -U mmuser

Check connection string:

docker exec mattermost env | grep MM_SQLSETTINGS_DATASOURCE

Expected: postgres://mmuser:BasicMM2026!@mattermost-db:5432/mattermost?sslmode=disable&connect_timeout=10

Dependencies

No dependencies on other local services.

Backup

Database Dump

docker exec mattermost-db pg_dump -U mmuser mattermost | gzip > ~/backups/mattermost-$(date +%Y%m%d-%H%M%S).sql.gz

Data Volumes

cd ~/system/services/mattermost
tar -czf ~/backups/mattermost-data-$(date +%Y%m%d-%H%M%S).tar.gz data/ config/ logs/ plugins/

Restore from Backup

# Stop service
cd ~/system/services/mattermost
docker compose down

# Restore database
gunzip -c ~/backups/mattermost-YYYYMMDD-HHMMSS.sql.gz | docker exec -i mattermost-db psql -U mmuser -d mattermost

# Restore data (if needed)
cd ~/system/services/mattermost
tar -xzf ~/backups/mattermost-data-YYYYMMDD-HHMMSS.tar.gz

# Start service
docker compose up -d

Configuration

Key Environment Variables

Full config: ~/system/services/mattermost/docker-compose.yml

Admin UI Config

Access: System Console (requires System Admin role)

Notes

Last updated: 2026-02-10 Maintained by: John (AI Director)

Planka Runbook

Runbook: Planka

Service Type: Kanban Board / Project Management Container: planka (ghcr.io/plankanban/planka:2.0.0-rc.4) Ports: 3100 (external) → 1337 (internal) External URL: https://boards.alai.no Database: PostgreSQL 15 (planka-db) Compose File: ~/system/services/planka/docker-compose.yml

Service Info

Planka is the visual project management tool for BasicAS Group. Kanban-style boards for task tracking.

Stack:

External Access:

Admin Access:

Status Check

Container Health

docker ps | grep planka

Expected output:

planka        Up X hours (healthy)
planka-db     Up X hours (healthy)

HTTP Check

curl -I http://localhost:3100

Expected: 200 OK or 302 Found

External Access Check

curl -I https://boards.alai.no

Expected: 200 OK or 302 Found

Database Check

docker exec planka-db psql -U postgres -d planka -c "SELECT count(*) FROM \"user\";"

Restart Procedure

Quick Restart (Container Only)

docker restart planka

Full Stack Restart (Container + Database)

cd ~/system/services/planka
docker compose down
docker compose up -d

Wait 30 seconds for healthcheck to pass, then verify:

docker ps | grep planka
curl -I http://localhost:3100

Troubleshooting

Problem: Container won't start

Check logs:

docker logs planka --tail 100

Common causes:

  1. Database not ready - wait 30s and retry
  2. Port 3100 already bound - check lsof -i :3100
  3. Volume permission issues - check docker volumes

Fix:

cd ~/system/services/planka
docker compose down
docker compose up -d planka-db
sleep 30
docker compose up -d planka

Problem: Login issues (can't sign in with admin credentials)

Check environment variables:

docker exec planka env | grep DEFAULT_ADMIN

Expected:

DEFAULT_ADMIN_EMAIL=john@alai.no
DEFAULT_ADMIN_PASSWORD=BasicAS2026!
DEFAULT_ADMIN_NAME=John AI
DEFAULT_ADMIN_USERNAME=john

If admin was changed in UI, default credentials won't work. Reset via database:

docker exec planka-db psql -U postgres -d planka -c "SELECT email, username FROM \"user\" WHERE \"isAdmin\" = true;"

Problem: 502 Bad Gateway (external access)

Check container is running:

docker ps | grep planka

Check Cloudflare tunnel:

cloudflared tunnel info boards

Check BASE_URL:

docker exec planka env | grep BASE_URL

Expected: BASE_URL=https://boards.alai.no

Problem: Database connection issues

Check database health:

docker exec planka-db pg_isready -U postgres -d planka

Check connection string:

docker exec planka env | grep DATABASE_URL

Expected: DATABASE_URL=postgresql://postgres@planka-db/planka

API Access

Planka has a REST API. Example:

Get Boards (requires auth token)

curl -H "Authorization: Bearer <TOKEN>" http://localhost:3100/api/boards

Get Token:

  1. Login via UI
  2. Inspect browser Network tab → find accessToken in response
  3. Or use user credentials to authenticate programmatically

Dependencies

No dependencies on other local services.

Backup

Database Dump

docker exec planka-db pg_dump -U postgres planka | gzip > ~/backups/planka-$(date +%Y%m%d-%H%M%S).sql.gz

Docker Volumes (includes file uploads)

docker run --rm -v planka-data:/data -v ~/backups:/backup alpine tar -czf /backup/planka-data-$(date +%Y%m%d-%H%M%S).tar.gz -C /data .
docker run --rm -v planka-db-data:/data -v ~/backups:/backup alpine tar -czf /backup/planka-db-data-$(date +%Y%m%d-%H%M%S).tar.gz -C /data .

Restore from Backup

# Stop service
cd ~/system/services/planka
docker compose down

# Restore database
gunzip -c ~/backups/planka-YYYYMMDD-HHMMSS.sql.gz | docker exec -i planka-db psql -U postgres -d planka

# Restore volumes (if needed)
docker run --rm -v planka-data:/data -v ~/backups:/backup alpine tar -xzf /backup/planka-data-YYYYMMDD-HHMMSS.tar.gz -C /data
docker run --rm -v planka-db-data:/data -v ~/backups:/backup alpine tar -xzf /backup/planka-db-data-YYYYMMDD-HHMMSS.tar.gz -C /data

# Start service
docker compose up -d

Configuration

Key Environment Variables

Full config: ~/system/services/planka/docker-compose.yml

Notes

Last updated: 2026-02-10 Maintained by: John (AI Director)

Documenso Runbook

Runbook: Documenso

Service Type: Document Signing Platform Container: documenso (documenso/documenso:latest) Ports: 3003 (external + internal) External URL: https://sign.alai.no Database: PostgreSQL 15 (documenso-db) Storage: MinIO (S3-compatible object storage) Compose File: ~/system/services/documenso/docker-compose.yml

Service Info

Documenso is the document signing platform for BasicAS Group. Used for NDAs, contracts, proposals.

Stack:

External Access:

Admin Access:

Status Check

Container Health

docker ps | grep documenso

Expected output:

documenso          Up X hours
documenso-db       Up X hours (healthy)
documenso-minio    Up X hours

Note: documenso-minio-setup exits after creating bucket (normal).

HTTP Check

curl -I http://localhost:3003

Expected: 200 OK or 307 Temporary Redirect

External Access Check

curl -I https://sign.alai.no

Expected: 200 OK or 307 Temporary Redirect

Database Check

docker exec documenso-db psql -U documenso_user -d documenso_db -c "SELECT count(*) FROM \"User\";"

(Use credentials from .env file)

MinIO Check

curl -I http://localhost:9002/minio/health/live

Expected: 200 OK

Restart Procedure

Quick Restart (Container Only)

docker restart documenso

Full Stack Restart (All Services)

cd ~/system/services/documenso
docker compose down
docker compose up -d

Wait 30-60 seconds for database healthcheck, then verify:

docker ps | grep documenso
curl -I http://localhost:3003

Troubleshooting

Problem: Container won't start

Check logs:

docker logs documenso --tail 100

Common causes:

  1. Database not ready - wait 30s and retry
  2. Port 3003 already bound - check lsof -i :3003
  3. Environment variables missing - check .env file
  4. MinIO not accessible - check minio container

Fix:

cd ~/system/services/documenso
docker compose down
docker compose up -d database minio
sleep 30
docker compose up -d documenso

Problem: Can't upload documents (500 error on upload)

Check MinIO is running:

docker ps | grep minio

Check MinIO bucket exists:

docker exec documenso-minio mc ls local/documenso

Expected: Bucket should exist (created by minio-setup).

Recreate bucket if missing:

docker exec documenso-minio mc mb local/documenso

Check Documenso S3 config:

docker exec documenso env | grep UPLOAD

Expected:

NEXT_PUBLIC_UPLOAD_TRANSPORT=s3
NEXT_PRIVATE_UPLOAD_ENDPOINT=http://host.docker.internal:9000
NEXT_PRIVATE_UPLOAD_BUCKET=documenso
NEXT_PRIVATE_UPLOAD_ACCESS_KEY_ID=documenso

Problem: Signature emails not sending

Check SMTP config:

docker exec documenso env | grep SMTP

Check .env file:

cd ~/system/services/documenso
grep SMTP .env

Expected:

NEXT_PRIVATE_SMTP_HOST=send.one.com
NEXT_PRIVATE_SMTP_PORT=465
NEXT_PRIVATE_SMTP_USERNAME=info@alai.no
NEXT_PRIVATE_SMTP_PASSWORD=<password>
NEXT_PRIVATE_SMTP_FROM_ADDRESS=info@alai.no

Test SMTP manually:

openssl s_client -connect send.one.com:465 -crlf

Problem: Database connection issues

Check database health:

docker exec documenso-db pg_isready -U documenso_user

Check connection string:

docker exec documenso env | grep DATABASE_URL

Expected: postgresql://documenso_user:<password>@database:5432/documenso_db

Problem: Signing fails (certificate error)

Check certificate exists:

ls -lh ~/system/services/documenso/certs/cert.p12

Check cert is mounted:

docker exec documenso ls -lh /opt/documenso/cert.p12

Check passphrase is set:

docker exec documenso env | grep SIGNING_PASSPHRASE

Webhook Integration

Documenso can send webhooks on document events (signed, completed, etc.).

Setup:

  1. Login to Documenso UI
  2. Go to Settings → Webhooks
  3. Add webhook URL (e.g., Mattermost incoming webhook)
  4. Select events (document.signed, document.completed)

Task #311: Integrate with Mattermost for signature notifications.

Dependencies

No dependencies on other local services.

Backup

Database Dump

docker exec documenso-db pg_dump -U documenso_user documenso_db | gzip > ~/backups/documenso-$(date +%Y%m%d-%H%M%S).sql.gz

MinIO Data (PDFs and files)

docker exec documenso-minio mc mirror local/documenso /tmp/documenso-backup
docker cp documenso-minio:/tmp/documenso-backup ~/backups/documenso-minio-$(date +%Y%m%d-%H%M%S)

Or use docker volume:

docker run --rm -v documenso_minio_data:/data -v ~/backups:/backup alpine tar -czf /backup/documenso-minio-$(date +%Y%m%d-%H%M%S).tar.gz -C /data .

Restore from Backup

# Stop service
cd ~/system/services/documenso
docker compose down

# Restore database
gunzip -c ~/backups/documenso-YYYYMMDD-HHMMSS.sql.gz | docker exec -i documenso-db psql -U documenso_user -d documenso_db

# Restore MinIO data
docker run --rm -v documenso_minio_data:/data -v ~/backups:/backup alpine tar -xzf /backup/documenso-minio-YYYYMMDD-HHMMSS.tar.gz -C /data

# Start service
docker compose up -d

Configuration

Key Environment Variables (.env file)

Security: .env file contains secrets - NOT in git, NOT in docker-compose.yml.

Full config: ~/system/services/documenso/.env

Notes

Last updated: 2026-02-10 Maintained by: John (AI Director)

Mission Control Dashboard Runbook

Runbook: Mission Control Dashboard

Service Type: Task Management Web UI Runtime: Node.js (Express) Port: 3030 (internal + LAN accessible) Internal URL: http://localhost:3030 LAN URL: http://192.168.68.61:3030 (mobile-friendly) Database: SQLite (~/system/databases/mission-control.db) LaunchAgent: com.john.mc-dashboard Source: ~/system/tools/mc-dashboard.js

Service Info

Mission Control Dashboard is the web UI for task management. Provides CRUD operations, priority management, status tracking, and team coordination.

Features:

CLI Alternative:

node ~/system/tools/mc.js list|add|start|done|pause|resume|block

Status Check

LaunchAgent Status

launchctl list | grep mc-dashboard

Expected output: PID shown (e.g., 12345 0 com.john.mc-dashboard)

If not running: - 0 com.john.mc-dashboard (no PID)

HTTP Check

curl -I http://localhost:3030

Expected: 200 OK

LAN Access Check (from another device)

curl -I http://192.168.68.61:3030

Expected: 200 OK

Database Check

sqlite3 ~/system/databases/mission-control.db "SELECT count(*) FROM tasks WHERE status = 'open';"

Restart Procedure

Stop Service

launchctl unload ~/Library/LaunchAgents/com.john.mc-dashboard.plist

Start Service

launchctl load ~/Library/LaunchAgents/com.john.mc-dashboard.plist

Restart (Stop + Start)

launchctl unload ~/Library/LaunchAgents/com.john.mc-dashboard.plist
launchctl load ~/Library/LaunchAgents/com.john.mc-dashboard.plist

Note: LaunchAgent auto-restarts on crash (KeepAlive=true).

View Logs

stdout (General logs)

tail -f ~/system/logs/mc-dashboard.log

stderr (Error logs)

tail -f ~/system/logs/mc-dashboard.err

Recent errors

tail -50 ~/system/logs/mc-dashboard.err

Troubleshooting

Problem: Dashboard won't start

Check LaunchAgent:

launchctl list | grep mc-dashboard

Check error log:

tail -50 ~/system/logs/mc-dashboard.err

Common causes:

  1. Port 3030 already bound - check lsof -i :3030
  2. Database locked - check for stale processes using SQLite
  3. Node.js not found - check which node
  4. Permission issues - check file ownership

Fix:

# Kill any process on port 3030
lsof -ti :3030 | xargs kill -9

# Restart
launchctl unload ~/Library/LaunchAgents/com.john.mc-dashboard.plist
launchctl load ~/Library/LaunchAgents/com.john.mc-dashboard.plist

Problem: Can't connect from mobile (LAN)

Check service is listening on all interfaces:

lsof -i :3030

Expected: *:3030 (listening on all IPs, not just 127.0.0.1)

Check firewall:

sudo /usr/libexec/ApplicationFirewall/socketfilterfw --getglobalstate

If firewall is on, allow Node.js:

sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /opt/homebrew/bin/node

Check Mac IP:

ipconfig getifaddr en0  # WiFi
ipconfig getifaddr en1  # Ethernet

Expected: 192.168.68.61 (or similar)

Problem: Tasks not updating (stale data)

Check database integrity:

sqlite3 ~/system/databases/mission-control.db "PRAGMA integrity_check;"

Expected: ok

Check last write:

ls -lh ~/system/databases/mission-control.db

Restart dashboard:

launchctl unload ~/Library/LaunchAgents/com.john.mc-dashboard.plist
launchctl load ~/Library/LaunchAgents/com.john.mc-dashboard.plist

Problem: 500 errors in UI

Check server logs:

tail -f ~/system/logs/mc-dashboard.log ~/system/logs/mc-dashboard.err

Check database:

sqlite3 ~/system/databases/mission-control.db "SELECT * FROM tasks LIMIT 1;"

Common causes:

  1. Database schema mismatch - migrate database
  2. Corrupted task data - fix in SQLite
  3. Node.js error - check stack trace in error log

CLI Integration

Mission Control has two interfaces:

  1. Dashboard (UI) - http://localhost:3030
  2. CLI - node ~/system/tools/mc.js

Both read/write the same SQLite database: ~/system/databases/mission-control.db

CLI Commands

# List tasks
node ~/system/tools/mc.js list
node ~/system/tools/mc.js list --owner john

# Start task (creates /tmp/mc-active-task)
node ~/system/tools/mc.js start <id>

# Complete task
node ~/system/tools/mc.js done <id> "outcome summary"

# Pause task (removes /tmp/mc-active-task)
node ~/system/tools/mc.js pause <id>

# Block task
node ~/system/tools/mc.js block <id> "blocker reason"

# Show full details
node ~/system/tools/mc.js show <id>

# Who's working on what
node ~/system/tools/mc.js active

Dependencies

Backup

Database Backup

cp ~/system/databases/mission-control.db ~/backups/mission-control-$(date +%Y%m%d-%H%M%S).db

Automated Backup (daily)

Add to crontab or LaunchAgent:

0 2 * * * cp ~/system/databases/mission-control.db ~/backups/mission-control-$(date +\%Y\%m\%d).db

Restore from Backup

# Stop dashboard
launchctl unload ~/Library/LaunchAgents/com.john.mc-dashboard.plist

# Restore database
cp ~/backups/mission-control-YYYYMMDD-HHMMSS.db ~/system/databases/mission-control.db

# Start dashboard
launchctl load ~/Library/LaunchAgents/com.john.mc-dashboard.plist

Configuration

LaunchAgent Plist

Path: ~/Library/LaunchAgents/com.john.mc-dashboard.plist

Key settings:

Application Config

Port: 3030 (hardcoded in mc-dashboard.js) Database: ~/system/databases/mission-control.db (hardcoded) Auto-refresh: 30 seconds (client-side)

To change port:

  1. Edit ~/system/tools/mc-dashboard.js
  2. Change const PORT = 3030; to desired port
  3. Restart LaunchAgent

Mission Control Session Worker

LaunchAgent: com.john.mc-session-worker Purpose: Background daemon for session-level task monitoring

Status check:

launchctl list | grep mc-session-worker

Notes

Last updated: 2026-02-10 Maintained by: John (AI Director)

Email System Runbook

Email System Runbook

TLDR

Overview

Centralized email system for ALAI/BasicAS. All outbound email goes through IMAP/SMTP (one.com + domeneshop), with a single audit database tracking everything.

Accounts

Account Email Provider Usage
john john@alai.no Migadu Primary business email
info info@alai.no Migadu General inquiries
alai john@alai.no Migadu ALAI branded/manual alias only; not scanned by email-agent.js because it is the same physical mailbox as john

Credentials stored in Vaultwarden. The runtime path (~/system/tools/mail-native.js) loads credentials via one per-process/cycle bw list items call and then serves all account configs from an in-memory map; do not reintroduce per-account sequential bw get item ... calls in daemon loops (MC #105900).

Runtime ingest note (MC #105906): ~/system/daemons/email-agent.js must use date-range IMAP search ({ since: sinceDate }) in both normal and legacy hot paths. Do not reintroduce unseen-only search ({ unseen: true } / { seen: false }): it misses messages already marked \\Seen by another client or by the john pass. The daemon also excludes the duplicate alai label from its scanned account array; mail-native.js may still keep the label for manual send/read compatibility.

How to Send Email

Option 1: MCP (from Claude session — PREFERRED)

mcp__email__email_send({
  account_name: "john",
  to: "client@example.com",
  subject: "Subject",
  body: "Body text",
  body_type: "html",          // or "plain"
  attachments: [{path: "/absolute/path/file.pdf"}]  // optional
})

Option 2: CLI (from scripts, daemons, agents)

node ~/system/tools/mail-native.js send \
  --to client@example.com \
  --subject "Subject" \
  --body "Body text" \
  --account john \
  --attachment /path/to/file.pdf   # optional, comma-separated for multiple

Option 3: Signing Emails (DocuSeal)

node ~/system/tools/send-signing-email.js send <template_id> '<signer_json>' --test

How to Read Email

MCP (preferred)

mcp__email__emails_find({account_name: "john", query: "invoice", limit: 10})
mcp__email__email_respond({email_id: "12345", body: "Reply text"})

CLI

node ~/system/tools/mail-native.js search "invoice" --account john --limit 20
node ~/system/tools/mail-native.js read <uid> --account john
node ~/system/tools/mail-native.js unread --account john
node ~/system/tools/mail-native.js reply <uid> --body "Reply text"
node ~/system/tools/mail-native.js forward <uid> --to other@email.com
node ~/system/tools/mail-native.js attachment <uid> --save /tmp/downloads

Email Audit (Single Source of Truth)

Database: ~/system/databases/email-audit.db

Every outbound email is logged here, regardless of send path:

Quick Commands

node ~/system/tools/email-audit.js recent               # Last 10 sent emails
node ~/system/tools/email-audit.js find "client name"    # Search all emails
node ~/system/tools/email-audit.js find "invoice" --days 30  # Last 30 days
node ~/system/tools/email-audit.js stats --days 7        # Stats by tool/account
node ~/system/tools/email-audit.js health                # System health check
node ~/system/tools/mail-native.js audit --days 30       # Audit from CLI
node ~/system/tools/mail-native.js sent --account john   # IMAP Sent folder

Architecture

Send paths:
  MCP (email_send/respond) ──┐
  mail-native.js CLI ────────┤──→ email-audit.db (single source of truth)
  send-signing-email.js ─────┤
  Hook (email-outbox-logger) ─┘ (safety net, dedup by message_id)

DEPRECATED Tools (DO NOT USE)

Tool Replacement
email.js mail-native.js
email-monitor.js MCP bridge
email-outbox.db email-audit.db
Inline SMTP scripts BLOCKED by bash-security-gate.py

Email-to-Task Integration

Automatic Task Creation

The email-agent daemon (~/system/daemons/email-agent.js) automatically creates MC tasks for ACTION emails:

Backlog Processing (MC #9269)

On daemon startup, email-agent processes ACTION emails that missed MC task creation:

SELECT id, message_id, account, from_addr, from_name, subject, date, action_needed, summary, priority
FROM emails
WHERE classification = 'ACTION' AND mc_task_id IS NULL
ORDER BY date DESC

How it works:

  1. Query runs once per daemon startup (before new email processing)
  2. For each backlog email: calls email-to-task.js with full context
  3. Extracts MC task ID from output (MC task #1234)
  4. Updates email record: UPDATE emails SET mc_task_id = ? WHERE id = ?
  5. Non-blocking: errors logged but don't crash daemon

Idempotency: Duplicate detection handled by mc.js (same title + <24h → link to existing task)

Removed 24h cutoff (2026-04-25):

Manual Backfill Procedure

If ACTION emails accumulate without MC tasks:

# 1. Check backlog count
sqlite3 ~/system/databases/email-inbox.db \
  "SELECT COUNT(*) FROM emails WHERE classification='ACTION' AND mc_task_id IS NULL;"

# 2. Restart daemon (triggers backlog processing)
launchctl kickstart -k gui/$(id -u)/com.john.email-agent

# 3. Verify log
tail -50 ~/system/logs/email-agent.log | grep -A10 "BACKLOG PROCESSING"

# 4. Check result
sqlite3 ~/system/databases/email-inbox.db \
  "SELECT COUNT(*) FROM emails WHERE classification='ACTION' AND mc_task_id IS NULL;"

Reference: /tmp/mc-9269-completion-report.md — 2026-04-25 backfill (78 emails → 69 MC tasks)

Email Tracker (Open/Click Tracking)

Service: com.john.email-tracker (LaunchAgent, KeepAlive) File: ~/system/tools/email-tracker.js Port: 3456 (127.0.0.1 only) Logs: ~/system/logs/email-tracker-stdout.log / email-tracker-stderr.log

Modes

node ~/system/tools/email-tracker.js              # server mode (default, daemon)
node ~/system/tools/email-tracker.js stats        # print DB counts, exit 0
node ~/system/tools/email-tracker.js tail         # stream new emails as log

Endpoints

Endpoint Description
GET /health {"ok":true} liveness check
GET /api/dashboard JSON stats: by_status, by_class, tracking events
GET /track/:emailId/open Log open event (returns 1x1 GIF)
GET /track/:emailId/click Log click event

DB Tables

Commands

# Quick stats
node ~/system/tools/email-tracker.js stats

# Live dashboard
curl -s http://127.0.0.1:3456/api/dashboard | jq .

# Reload daemon (after file changes or crash)
launchctl kickstart -k gui/$(id -u)/com.john.email-tracker

# Check status
launchctl print gui/$(id -u)/com.john.email-tracker | grep -E "state|pid"

Troubleshooting

Email not in audit

  1. Check node email-audit.js recent — is it really missing?
  2. Check MCP bridge log: tail ~/system/logs/email-mcp-bridge.log
  3. Check mail-native log: tail ~/system/logs/mail-native.log
  4. Run node email-audit.js health — any warnings?

SMTP connection fails

  1. Check vault: bw get item "Migadu — john@alai.no" --session $(cat /tmp/bw-session) | jq .login.username
  2. Test: node mail-native.js test --account john
  3. one.com rate limits: wait 5 min, retry

Attachments not working

  1. Verify file exists: ls -la /path/to/file
  2. Use absolute paths only
  3. Max attachment size: ~25MB (one.com limit)
  4. CLI: --attachment /path/file1.pdf,/path/file2.pdf (comma-separated)
  5. MCP: attachments: [{path: "/abs/path"}] (array of objects)

ACTION email has no MC task

Symptoms: Email classified as ACTION but mc_task_id IS NULL in database

Diagnosis:

# Check specific email
sqlite3 ~/system/databases/email-inbox.db \
  "SELECT id, from_addr, subject, classification, mc_task_id FROM emails WHERE id = <email_id>;"

# Check backlog count
sqlite3 ~/system/databases/email-inbox.db \
  "SELECT COUNT(*) FROM emails WHERE classification='ACTION' AND mc_task_id IS NULL;"

Fix:

  1. Check daemon is running: launchctl list | grep email-agent
  2. Check daemon log: tail -100 ~/system/logs/email-agent.log
  3. Trigger backlog processing: launchctl kickstart -k gui/$(id -u)/com.john.email-agent
  4. If still NULL after daemon cycle:
    • Manual task creation: node ~/system/tools/email-to-task.js --from "..." --subject "..." --message-id "..."
    • Update email record: sqlite3 ~/system/databases/email-inbox.db "UPDATE emails SET mc_task_id = <task_id> WHERE id = <email_id>;"

Attachment fetch hangs / "Socket timeout" (email-attachment-fetcher.js)

Symptoms: email-attachment-fetcher.js <db_id> dies after ~60-120s with IMAP client error: Socket timeout / Connection not available, even though the mail exists on the server.

Root causes (both fixed 2026-08-02, MC #106645):

  1. Wide envelope scan: RFC2822 Message-ID lookup used a fixed 30-day SINCE window — hundreds of envelopes on a large inbox blew the socket timeout. Now: narrow window from the DB date column (±2 days, SINCE+BEFORE) is tried first; the 30-day scan remains only as fallback when the date hint is missing or wrong.
  2. imapflow deadlock footgun: fetchOne(source) was called inside an active for await client.fetch(...) envelope stream. In imapflow a new command queues behind the running FETCH, whose generator is paused awaiting the loop body — both wait forever until socket timeout. Rule: never issue IMAP commands inside a fetch stream; record the matching UID, let the chunk drain, fetch source after.

Diagnosis:

# stderr shows which window fired and how many candidates:
node ~/system/tools/email-attachment-fetcher.js <db_id>
# "SINCE window 2026-07-29..2026-08-02: 67 candidates" = narrow path OK
# Hundreds of candidates = date hint missing → check emails.date for that row

Reference repro: mail #15176 (hjk.hr, 4 PDF attachments) — before: 2x socket timeout; after: fetch_ms≈4.6s. Evidence: ~/system/evidence/106645/.

Remote pristup na Mac Studio

Remote pristup na Mac Studio

Pregled

Mac Studio (192.168.68.61) je dostupan na dva načina:

Lokalni pristup (kuća/kancelarija)

ssh makinja@192.168.68.61

Ili sa SSH config alijasom:

ssh studio

SSH config na Air-u (~/.ssh/config):

Host studio
    HostName 192.168.68.61
    User makinja
    ServerAliveInterval 30
    ServerAliveCountMax 120
    TCPKeepAlive yes
    RequestTTY yes

Remote pristup (izvana)

Koristi Cloudflare Tunnel sa Zero Trust Access zaštitom.

Preduvjeti (jednom na Air-u):

brew install cloudflared

SSH config na Air-u (~/.ssh/config):

Host studio-remote
    HostName ssh.basicconsulting.no
    User makinja
    ProxyCommand cloudflared access ssh --hostname %h
    RequestTTY yes

Konekcija:

ssh studio-remote

Browser se otvori → logiraš se sa alem@alai.no → spojen si na Studio.

Dozvoljeni emailovi:

tmux (persistent sesije)

Kad Air zaspi, SSH konekcija umre. tmux drži sesiju živom na Studiju.

Automatski (konfigurirano u ~/.zshrc na Studiju):

SSH konekcija automatski attach-a tmux sesiju kad je terminal dostupan.

Ručno:

# Nova sesija
tmux new -s main

# Attach na postojeću
tmux attach -t main

# Detach (izlaz bez zatvaranja)
Ctrl+A, D

# Lista sesija
tmux list-sessions

tmux prečice:

Komanda Akcija
Ctrl+A, D Detach (izađi, sesija živi)
Ctrl+A, | Split vertikalno
Ctrl+A, - Split horizontalno
Ctrl+A, h/j/k/l Navigacija između panela

Security

Mjera Status
PasswordAuthentication no (samo ključevi)
AllowTcpForwarding no
AuthenticationMethods publickey
AllowUsers makinja
MaxAuthTries 3
Cloudflare Access alem@alai.no + backup emailovi
Vaultwarden 2FA uključen

Troubleshooting

SSH ne radi lokalno:

# Provjeri da je SSH upaljen
sudo systemsetup -getremotelogin

# Restart SSH
sudo launchctl kickstart -k system/com.openssh.sshd

Remote ne radi:

# Provjeri cloudflared na Studiju
launchctl list com.john.cloudflared

# Restart tunnel
launchctl kickstart -k gui/$(id -u)/com.john.cloudflared

# Provjeri DNS
dig ssh.basicconsulting.no

tmux ne radi:

# Koristi -t flag za SSH
ssh -t makinja@192.168.68.61

# Provjeri sesije
tmux list-sessions

Zadnje ažuriranje: 2026-02-24 Kreirao: John (system audit + security hardening)

Remote Access — VNC Studio via Cloudflare

Remote Access — VNC Studio via Cloudflare

Prerequisites

Architecture

Studio: macOS Screen Sharing (VNC :5900)
    → Cloudflare Tunnel
    → vnc.basicconsulting.no
    → Air: cloudflared TCP proxy
    → localhost:5901
    → Finder VNC client

Studio Setup (One-time — already done)

Cloudflared config (~/.cloudflared/config.yml) includes:

- hostname: vnc.basicconsulting.no
  service: tcp://localhost:5900

Tunnel runs as a LaunchAgent:

# Check tunnel status
launchctl list com.john.cloudflared

# Restart if needed
launchctl kickstart -k gui/$(id -u)/com.john.cloudflared

How to Connect from Air

  1. Open Terminal on Air

  2. Start the local TCP proxy:

cloudflared access tcp --hostname vnc.basicconsulting.no --url localhost:5901

  1. Open Finder → Go → Connect to Server (Cmd+K)

  2. Enter:

vnc://localhost:5901
  1. Enter the VNC password when prompted

Keep the Terminal window open — the cloudflared proxy must stay running for the session.

Troubleshooting

Connection fails:

# On Studio — verify tunnel is running
ps aux | grep cloudflared

# Restart tunnel on Studio
launchctl kickstart -k gui/$(id -u)/com.john.cloudflared

Lag / slow display:

Display is at 2560x1440 (5K scaled). Reduce resolution temporarily to improve performance:

# Reduce to 1080p (lower lag)
displayplacer "id:D8EAE737-E4F0-42D1-9AD0-C39CDD691C67 res:1920x1080 hz:60 color_depth:8 scaling:on enabled:true"

# Restore original resolution
displayplacer "id:D8EAE737-E4F0-42D1-9AD0-C39CDD691C67 res:2560x1440 hz:60 color_depth:8 scaling:on enabled:true"

Port conflict on Air:

If port 5901 is already in use:

lsof -i :5901

Use a different port (e.g. 5902) and update the vnc:// address accordingly.

noVNC (Browser fallback — NOT recommended)

Tested and unusable due to lag on 5K display. Documented here for reference only.

Install:

pip3 install --break-system-packages websockify
git clone https://github.com/novnc/noVNC.git ~/novnc

Run (on Studio):

websockify --web ~/novnc 6080 localhost:5900

Access: https://remote.basicconsulting.no/vnc.html?resize=scale

Verdict: Unusable lag at 5K resolution. Use the cloudflared TCP method above.

Last updated: 2026-02-24 Created by: John — post VNC remote access session

Rad bez Claude Code — Emergency Mode

Rad bez Claude Code — Emergency Mode

Ovaj dokument opisuje kako koristiti ALAI sistem kada Claude Code (CC) nije dostupan — bilo zbog API limita, održavanja, ili bilo kojeg drugog razloga.

Zadnja izmjena: 2026-02-26 | MC Task: #2074 | Status: Phase 1 Complete

Pregled: Šta radi, šta ne radi

Radi BEZ Claude Code (~60% sistema)

Komponenta Kako pokrenuti Napomena
Mission Control node ~/system/tools/mc.js list Potpuno nezavisan od CC
HiveMind node ~/system/agents/hivemind/hivemind.js query "text" 16K+ entries, lokalna baza
RAG/Knowledge mcp__rag__rag_query ili retrieval-orchestrator Lokalni cache + Ollama
Email Emergency REPL: email list Email agent na Ollami
Ollama AI node ~/system/tools/ollama-engine.js generate "prompt" Lokalni modeli
Ollama Tool Agent node ~/system/tools/ollama-tool-agent.js --task "..." Cita I pise fajlove (sa safety)
Chain Runner node ~/system/tools/chain-runner.js run <chain> "input" YAML chain sa Ollama agentima
Daemoni Svi 13 daemona rade nezavisno ops-watchdog jedini koristi CC
BookStack http://localhost:6875 (ili docs.basicconsulting.no) Wiki radi nezavisno
Dashboard http://localhost:3030 MC dashboard u browseru
Slack node ~/system/tools/slack.js send <channel> "msg" Radi nezavisno

NE radi bez Claude Code (~40%)

Komponenta Zašto Alternativa
CC Subagenti (builder/validator) Zahtijeva claude CLI Ollama tool agent za jednostavne taskove
Hooks (54 Python hookova) CC ih triggera automatski Rucno pozvati write-guard.js za safety
Skills (93 skilla) CC prompt template sistem Rucno pokrenuti chain ili Ollama agent
Agent Teams (TeamCreate) CC inter-agent komunikacija Sekvencionalno kroz Ollama agente
Interaktivna sesija Multi-turn kontekst Emergency REPL (single-turn)
ops-watchdog daemon Hardcoded claude CLI Planirano za Phase 2 migraciju

Quick Start: Emergency Boot

bash ~/system/tools/emergency-boot.sh

Ovo pokrece:

  1. Ollama health check (provjera modela)
  2. System status (MC taskovi, HiveMind, email)
  3. Emergency REPL — interaktivni shell

Emergency REPL — Komande

Kad se REPL pokrene, dobijas john> prompt.

Task Management

mc list                      # Lista otvorenih taskova
mc show <id>                 # Detalji taska
mc add "Naslov taska"        # Dodaj novi task
mc start <id>                # Zapocni rad
mc done <id> "Outcome"       # Zavrsi task
mc stats                     # Statistika

Knowledge Base

hm query "search text"       # Pretrazi HiveMind (16K+ entries)
hm post john task "text"     # Dodaj u HiveMind
hm status                    # Status baze

Email

email list                   # Lista emailova (john account)
email list info              # Lista emailova (info account)
email read <id>              # Procitaj email

AI (Ollama)

ask "Koji su otvoreni taskovi za Alema?"    # Jedan prompt -> Ollama odgovor
agent "Nadji sve fajlove koji importuju X"  # Multi-turn agent sa tools
agent "Napisi helper funkciju za Y"         # Agent moze i PISATI fajlove
chain <chain-name> "input"                  # Pokreni YAML chain

Sistem

status                       # Health check (Ollama, MC, HiveMind, CC)
help                         # Lista svih komandi
exit                         # Izlaz

Ollama Write Capability — Safety

Ollama agent sada moze pisati fajlove, ali sa strogim safety stackom:

Shadow Mode (aktivno prvu sedmicu)

Svi Ollama write-ovi idu u ~/system/backups/ollama-writes/.pending/ umjesto na pravu destinaciju. Moras rucno pregledati i odobriti.

Da iskljucis shadow mode (nakon provjere):

// ~/system/tools/config/ollama-write-config.json
{ "shadowMode": false }

Path Whitelist — Ollama MOZE pisati u:

Path Denylist (BLOKIRANO)

Audit Log

Svaki Ollama write se logira u: ~/system/logs/ollama-writes.jsonl

Secret Detection

Write-guard automatski skenira content za API kljuceve, passworde, tokene, private keys, database URL-ove. Ako detektuje — blokira write.

Provider Abstraction

Novi ~/system/lib/provider.js unificira pristup AI providerima:

const { Provider } = require('~/system/lib/provider');

// Auto — bira najjeftinijeg dostupnog (Ollama > Anthropic API > CC)
const p = await Provider.resolve('auto');
const result = await p.complete('prompt');

// Forsiraj Ollamu
const ollama = await Provider.resolve('ollama');

// Forsiraj Claude CLI
const cc = await Provider.resolve('claude');

Dostupni provideri

Provider Cijena Kada
Ollama Besplatno (lokalno) Default za auto, validatore, research
Anthropic API API cijena Kad treba Claude kvalitet bez CC overhead-a
Claude CLI CC cijena Kompleksni buildovi, multi-file taskovi

Test dostupnosti: node ~/system/lib/provider.js test

Tipicni Scenariji

Scenarij 1: CC je pao, trebam zavrsiti task

bash ~/system/tools/emergency-boot.sh
# U REPL-u:
mc list                          # Vidi sta je otvoreno
mc start 1234                    # Zapocni task
agent "Implement function X in ~/projects/Y/file.js"
mc done 1234 "Completed via emergency mode"

Scenarij 2: Trebam provjeriti emailove

bash ~/system/tools/emergency-boot.sh
email list john
email list info
email read 123

Scenarij 3: Trebam pregledati HiveMind/znanje

bash ~/system/tools/emergency-boot.sh
hm query "invoice Knowit"
hm query "NordFit deployment"

Scenarij 4: Quick AI pitanje

bash ~/system/tools/emergency-boot.sh
ask "Summarize the current status of task 2074"

Arhitektura

+--------------------------------------------+
|           INTERACTIVE LAYER                |
|  CC Session (primary) | Emergency REPL     |
+--------------------------------------------+
|           PROVIDER LAYER                   |
|  Provider.resolve() -> Claude|Ollama|API   |
+--------------------------------------------+
|           TOOL LAYER (portable)            |
|  MC|HiveMind|RAG|Email|Slack|BookStack     |
+--------------------------------------------+
|           STORAGE LAYER                    |
|  SQLite (tasks, leads) | JSONL (logs)      |
|  Markdown (context) | YAML (chains)        |
+--------------------------------------------+

Fajlovi — Phase 1 Deliverables

Fajl Opis
~/system/lib/provider.js Unified AI provider (Claude, Ollama, Anthropic API)
~/system/lib/write-guard.js Write safety (whitelist, secrets, backup, shadow)
~/system/tools/config/ollama-write-config.json Config za write-guard
~/system/tools/emergency-boot.sh Emergency mode launcher
~/system/tools/emergency-repl.js Interactive REPL bez CC
~/system/tools/ollama-tool-agent.js Extended sa write_file + edit_file
~/system/tools/agent-orchestrator.js Provider-aware worker spawning
~/system/tools/chain-runner.js Provider execution path
~/system/config/tier-routing.json Fallback chains per role
~/system/logs/ollama-writes.jsonl Audit log za Ollama writes

Phase 2 (Planirano)

Pitanja? Pokreni emergency REPL i pitaj: ask "How do I..."

Baikal CalDAV Runbook

Service: Baikal CalDAV

Label: Docker container baikal + LaunchAgent com.john.calendar-bridge Tier: P2 (Business) Port: 5232 (local), calendar.basicconsulting.no (public via Cloudflare)

What It Does

Self-hosted CalDAV server for ALAI Business calendar. Alem syncs from iPhone/MacBook via native Calendar app. calendar-bridge.js daemon scans emails every 5min, detects meeting invites, forwards to alem@alai.no, and creates CalDAV events.

Architecture

Email (john@) → email-agent.js → calendar-bridge.js → Baikal CalDAV → Alem iPhone/Mac
                                       ↓
                               mail-native.js forward → alem@alai.no

Components

Component Location Type
Baikal server ~/system/services/baikal/docker-compose.yml Docker
calendar-bridge.js ~/system/tools/calendar-bridge.js Tool + Daemon
LaunchAgent ~/Library/LaunchAgents/com.john.calendar-bridge.plist Daemon (5min)
Cloudflare tunnel calendar.basicconsulting.no → localhost:5232 Tunnel
Credentials Vaultwarden → "Baikal CalDAV" Vault
Calendar "ALAI Business" (CalDAV user: alem) CalDAV
Data ~/system/services/baikal/data/ Persistent volume

Dependencies

Health Check

# Quick check
node ~/system/tools/calendar-bridge.js test

# Docker container
docker ps --filter name=baikal

# CalDAV endpoint
curl -s -o /dev/null -w "%{http_code}" http://localhost:5232/dav.php/

# Public URL (expect 401 = auth required = healthy)
curl -s -o /dev/null -w "%{http_code}" https://calendar.basicconsulting.no/dav.php/

# List events
node ~/system/tools/calendar-bridge.js list

Common Failures & Fixes

Failure 1: Baikal container down

Symptoms: calendar-bridge.js test fails, CalDAV 502/connection refused Fix:

cd ~/system/services/baikal && docker compose up -d

Failure 2: Cloudflare tunnel not routing

Symptoms: Public URL returns 404 or timeout, local URL works fine Fix:

# Check config includes calendar entry
grep calendar ~/.cloudflared/config.yml
# Restart tunnel
launchctl kickstart -k gui/$(id -u)/com.john.cloudflared

Failure 3: Calendar-bridge scan finds nothing

Symptoms: Meeting invites arrive but no events created, no forwards Check:

# Check daemon is running
launchctl list | grep calendar-bridge
# Check logs
tail -50 ~/system/logs/calendar-bridge.log
# Check state file
cat ~/system/logs/calendar-bridge-state.json
# Manual scan with verbose
node ~/system/tools/calendar-bridge.js scan --verbose

Failure 4: Alem can't sync from iPhone

Symptoms: iPhone Calendar shows error, events not showing Check:

  1. Verify credentials in Vault: node ~/system/tools/vault.js get "Baikal CalDAV"
  2. Test public CalDAV endpoint (should return 401, not 502/404)
  3. iPhone settings: Server = calendar.basicconsulting.no/dav.php/principals/alem

Failure 5: Authentication failure

Symptoms: 401 with correct password Fix: Password might be out of sync. Re-hash in Baikal DB:

NEW_PASS=$(bw get password "Baikal CalDAV" --session $(cat /tmp/bw-session))
DIGEST=$(printf "alem:BaikalDAV:$NEW_PASS" | md5)
docker exec baikal sqlite3 /var/www/baikal/Specific/db/db.sqlite \
  "UPDATE users SET digesta1='$DIGEST' WHERE username='alem';"

Restart Procedure

# Restart Baikal
cd ~/system/services/baikal && docker compose restart

# Restart calendar-bridge daemon
launchctl kickstart -k gui/$(id -u)/com.john.calendar-bridge

Backup

MC Task

Created: #3029 (Deploy), #3035 (Documentation + Watchdog)

Infrastructure

Infrastructure runbooks: daemons, email, backups, monitoring

Infrastructure

Email Agent Runbook

Email Agent Runbook

Service: Email Agent Daemon
Location: ~/system/daemons/email-agent.js
LaunchAgent: com.john.email-agent
Interval: Every 5 minutes (300s)
Last Updated: 2026-04-15

1. Architecture

What It Does

The Email Agent is a 24/7 daemon that:

Accounts Monitored

Account Key Email Address Bitwarden Vault Name
johnjohn@basicconsulting.noEmail - john@basicconsulting.no
infoinfo@basicconsulting.noEmail - info@basicconsulting.no
alaijohn@alai.noEmail - john@alai.no
alemalem@alai.noEmail - alem@alai.no
devdev@alai.noEmail - dev@alai.no
gmailalembasic@gmail.comEmail - alembasic@gmail.com

Classification Pipeline

  1. VIP Bypass: Emails from CEO/family → forced to ACTION/high, label: CEO FORWARD
  2. Quick Filter: Pattern-based detection for OWN emails and known SPAM
  3. Ollama Classification: Remaining emails sent to local llama3.1:8b model
  4. Circuit Breaker: Falls back to pattern heuristics if Ollama is down (3 failure threshold)

VIP Senders (CEO Bypass List)

Emails from these addresses bypass all filters and are always classified as ACTION/high with label CEO FORWARD:

Transport: Himalaya Adapter

The daemon uses ~/system/tools/himalaya-adapter.js, which wraps the Rust-based himalaya CLI (/opt/homebrew/bin/himalaya).

Config: ~/.config/himalaya/config.toml — all 6 accounts configured.

2. Credentials

Bitwarden Storage

All email accounts are stored in Bitwarden with vault item names following the pattern: Email - <address>.

Gmail Account (Special Configuration)

The Gmail account (alembasic@gmail.com) uses App Password authentication (not the regular Google account password).

Bitwarden Item: Email - alembasic@gmail.com
Custom Fields in Vault:

Himalaya Config

File: ~/.config/himalaya/config.toml

Contains 6 account blocks with IMAP/SMTP settings. Credentials are loaded from Bitwarden at runtime via mail-native.js.

3. How to Verify

Is the Daemon Running?

launchctl list | grep email-agent
# Expected output: PID + exit status 0
# Example: 12345  0  com.john.email-agent

Last Heartbeat (Should Be < 10 Minutes Ago)

cat ~/system/logs/email-agent-heartbeat.txt
# Shows timestamp of last successful run

Recent Activity Log

tail -20 ~/system/logs/email-agent-launchd.log
# Should show recent classification activity like:
# {"timestamp":"2026-04-15T13:49:06.450Z","service":"email-agent","level":"info","message":"Classifying via Ollama: ..."}

Pending Emails (Email Inbox Tool)

node ~/system/tools/email-inbox.js pending
# Lists emails waiting for classification or action

Daemon Status (Full Details)

launchctl print gui/$(id -u)/com.john.email-agent
# Shows full launchd status, last run time, exit codes

4. Troubleshooting

Problem: Daemon Dead (MODULE_NOT_FOUND Error)

Symptom:

tail -20 ~/system/logs/email-agent-launchd-error.log
# Shows: Error: Cannot find module '~/system/tools/himalaya-adapter'

Root Cause: The himalaya-adapter.js file was accidentally archived or deleted.

Fix:

  1. Verify the file exists: ls -lh ~/system/tools/himalaya-adapter.js
  2. If missing, restore from ~/system/tools/archive/ or Git history
  3. Restart the daemon: 
    launchctl unload ~/Library/LaunchAgents/com.john.email-agent.plist
launchctl load ~/Library/LaunchAgents/com.john.email-agent.plist
  4. Verify restart: launchctl list | grep email-agent

Problem: Gmail "Unknown Account" Error

Symptom:

Error: Unknown account: gmail. Available: john, info, alai, alem, dev

Root Cause: The gmail key is missing from the VAULT_NAMES object in ~/system/tools/mail-native.js.

Fix:

  1. Open ~/system/tools/mail-native.js
  2. Locate the VAULT_NAMES object (around line 20)
  3. Add the gmail entry: 
    const VAULT_NAMES = {
  john: 'Email - john@basicconsulting.no',
  info: 'Email - info@basicconsulting.no',
  alai: 'Email - john@alai.no',
  alem: 'Email - alem@alai.no',
  dev: 'Email - dev@alai.no',
  gmail: 'Email - alembasic@gmail.com'  // Add this line
};
  4. Save and reload daemon

Problem: Gmail Hanging Daemon (High CPU/Memory)

Symptom:

Root Cause: Gmail IMAP fetch is hanging indefinitely, causing overlapping daemon instances.

Fix:

  1. Identify stuck process: 
    ps aux | grep email-agent
  2. Kill the stuck process gracefully: 
    kill -QUIT <PID>
# Or if unresponsive:
kill -9 <PID>
  3. Unload and reload daemon: 
    launchctl unload ~/Library/LaunchAgents/com.john.email-agent.plist
launchctl load ~/Library/LaunchAgents/com.john.email-agent.plist

Problem: Vault Credentials Unavailable (Circuit Breaker Triggered)

Symptom:

Error: Bitwarden session not available
# Or: Circuit breaker OPEN for account: john

Root Cause: Bitwarden CLI session expired or /tmp/bw-session is empty.

Fix:

  1. Check session file: 
    cat /tmp/bw-session
# Should contain a session token string
  2. If empty, unlock Bitwarden and regenerate session: 
    bw unlock --raw > /tmp/bw-session
# Enter master password when prompted
  3. Verify session works: 
    bw get item "Email - john@basicconsulting.no" --session $(cat /tmp/bw-session)
  4. Circuit breaker will reset automatically on next successful run (backoff resets after threshold period)

Problem: Alem's Emails Not Showing as ACTION

Symptom: Emails from CEO are classified as INFO or SPAM instead of ACTION/high.

Root Cause: VIP_SENDERS list is incomplete or outdated.

Fix:

  1. Open ~/system/daemons/email-agent.js
  2. Locate the VIP_SENDERS array (around line 92)
  3. Ensure all Alem's addresses are present: 
    const VIP_SENDERS = [
  'alem@alai.no',
  'alem@basicconsulting.no',
  'alem.basic@gmail.com',
  'alembasic@gmail.com',
  'sibilabasic@gmail.com',
  'riadbasic007@gmail.com'
];
  4. Save and reload daemon

Problem: Ollama Circuit Breaker Open (Fallback Mode)

Symptom:

WARN: Ollama circuit breaker OPEN — using pattern heuristic

Root Cause: Ollama service is down or unresponsive (3+ consecutive failures).

Fix:

  1. Check Ollama service: 
    curl http://localhost:11434/api/tags
# Should return JSON list of models
  2. If unresponsive, restart Ollama: 
    brew services restart ollama
# Or manually:
ollama serve
  3. Circuit breaker will auto-reset after backoff period (starts at 10s, max 5 minutes)
  4. Emails will still be processed using pattern-based heuristics during circuit breaker OPEN state

5. Gmail App Password Setup

If the Gmail App Password needs to be regenerated (e.g., after credential rotation or security incident):

  1. Go to https://myaccount.google.com/apppasswords (must be logged in as alembasic@gmail.com)
  2. Click Generate
  3. Select app: Mail
  4. Select device: Mac (or custom name like "IMAP Daemon")
  5. Copy the 16-character App Password (no spaces)
  6. Update Bitwarden: 
    bw get item "Email - alembasic@gmail.com" --session $(cat /tmp/bw-session) | \
  jq '.login.password = "<NEW_APP_PASSWORD>"' | \
  bw encode | \
  bw edit item $(bw get item "Email - alembasic@gmail.com" --session $(cat /tmp/bw-session) | jq -r .id) --session $(cat /tmp/bw-session)
    Or update manually via Bitwarden web vault.
  7. Reload daemon: 
    launchctl unload ~/Library/LaunchAgents/com.john.email-agent.plist
launchctl load ~/Library/LaunchAgents/com.john.email-agent.plist

6. Key Files and Locations

File Purpose
~/system/daemons/email-agent.jsMain daemon script
~/system/tools/mail-native.jsVAULT_NAMES map + credential loader
~/system/tools/himalaya-adapter.jsHimalaya CLI wrapper (IMAP/SMTP)
~/.config/himalaya/config.tomlHimalaya account configuration
~/Library/LaunchAgents/com.john.email-agent.plistLaunchAgent config (5-minute interval)
~/system/logs/email-agent-launchd.logDaemon stdout log
~/system/logs/email-agent-launchd-error.logDaemon stderr log
~/system/logs/email-agent-heartbeat.txtLast successful run timestamp
~/system/logs/email-triage-results.jsonlJSONL log of all classifications
/tmp/bw-sessionBitwarden CLI session token

7. Escalation

If the daemon is down for > 30 minutes and troubleshooting steps do not resolve:

  1. Check email-agent-launchd-error.log for stack traces
  2. Capture full logs: 
    tail -100 ~/system/logs/email-agent-launchd.log > /tmp/email-agent-debug.log
tail -100 ~/system/logs/email-agent-launchd-error.log >> /tmp/email-agent-debug.log
launchctl print gui/$(id -u)/com.john.email-agent >> /tmp/email-agent-debug.log
  3. Slack alert to #ops: 
    node ~/system/tools/slack.js send ops "@john Email Agent daemon DOWN for 30+ minutes. Logs: /tmp/email-agent-debug.log"
  4. Fallback: manually check inboxes via webmail until daemon is restored

Document Status: ✅ Production
Owner: John (primary agent)
Last Incident: 2026-02-25 — MODULE_NOT_FOUND (himalaya-adapter archived)
Last Review: 2026-04-15

Infrastructure

Runbook: LightRAG ingest LaunchAgent fix (MC #10286)

Overview

This runbook documents the investigation and fix applied to three LightRAG-related LaunchAgents on the ALAI Mac Studio host in MC #10286. The fix was validated by Proveo (Angie Jones) with a PARTIAL verdict: 3 PASS, 1 PARTIAL (AC3), 1 FAIL (AC4 — same-day unverifiable). CF Access root cause is tracked separately in MC #10298.

1. Symptom — How to Detect This Failure

These signals indicate the com.alai.lightrag-outbox-ingest LaunchAgent is failing silently:

2. Root Cause

The primary failure was in com.alai.lightrag-outbox-ingest:

Workaround applied: Changed LIGHTRAG_URL to http://localhost:9621 in the plist. The CF Access token 302 root cause (why the local host receives a redirect instead of being authorized) is tracked in MC #10298 (priority: M).

The other two daemons were not functionally broken:

3. Fix Procedure

Preconditions: You have shell access to the Mac Studio host. LightRAG is running locally on port 9621.

Step 1: Verify current plist URL

grep -A1 "LIGHTRAG_URL" ~/Library/LaunchAgents/com.alai.lightrag-outbox-ingest.plist

If the value is https://lightrag.alai.no, proceed. If already http://localhost:9621, skip to Step 4.

Step 2: Edit the plist

# Open in editor — change the LIGHTRAG_URL string value:
# FROM: https://lightrag.alai.no
# TO:   http://localhost:9621
nano ~/Library/LaunchAgents/com.alai.lightrag-outbox-ingest.plist

The relevant section in the plist:

<key>LIGHTRAG_URL</key><string>http://localhost:9621</string>

Step 3: Unload all 3 lightrag plists

launchctl unload ~/Library/LaunchAgents/com.alai.lightrag-outbox-ingest.plist
launchctl unload ~/Library/LaunchAgents/com.alai.lightrag-backup.plist
launchctl unload ~/Library/LaunchAgents/com.john.lightrag-monitor.plist

Step 4: Reload all 3 lightrag plists

launchctl load -w ~/Library/LaunchAgents/com.alai.lightrag-outbox-ingest.plist
launchctl load -w ~/Library/LaunchAgents/com.alai.lightrag-backup.plist
launchctl load -w ~/Library/LaunchAgents/com.john.lightrag-monitor.plist

Step 5: Drain the outbox manually (if backlog exists)

node ~/system/tools/lightrag-outbox-ingest.js

The script is idempotent — it uses outbox-ingest.sqlite with correlation_id as PRIMARY KEY dedup gate. Running it multiple times is safe. Expected output when backlog is cleared: processed: 0, skipped: N, failed: 0.

Step 6: Kickstart the ingest daemon to verify immediate fire

launchctl kickstart -k gui/$(id -u)/com.alai.lightrag-outbox-ingest

Check the log immediately after:

tail -20 ~/system/logs/lightrag-outbox-ingest.log

Expected: A [ingest] DONE line with exit success.

Step 7: Confirm watchdog detects healthy state

bash ~/bin/daemon-fleet-watchdog.sh 2>&1 | grep lightrag

Expected: All 3 labels in calendar_ok or calendar_ok state. No calendar_err_* or not_loaded transitions.

4. Verification Commands

# 1. All 3 plists loaded with LastExitStatus=0
launchctl list | grep lightrag

# 2. Checkpoint DB row count (should match mc-task-outcomes.jsonl line count)
sqlite3 ~/system/state/outbox-ingest.sqlite "SELECT count(*) FROM processed"

# 3. Most recent ingest timestamp
sqlite3 ~/system/state/outbox-ingest.sqlite "SELECT MAX(ingested_at) FROM processed"

# 4. LightRAG pipeline health
curl http://localhost:9621/documents/pipeline_status

# 5. LightRAG document total count
curl http://localhost:9621/documents | jq .total

# 6. Outbox log last run summary
grep "DONE" ~/system/logs/lightrag-outbox-ingest.log | tail -5

# 7. Watchdog recent transitions for lightrag
grep lightrag ~/system/logs/daemon-fleet-watchdog.log | tail -20

5. Known Limitations

6. Watchdog Coverage

The daemon-fleet-watchdog at ~/bin/daemon-fleet-watchdog.sh covers all 3 LightRAG plists via its glob at line 39:

for plist in "$HOME"/Library/LaunchAgents/com.{alai,john}.*.plist

This glob automatically includes any new LightRAG LaunchAgents matching the pattern without code changes. The watchdog runs every 15 minutes via com.alai.daemon-fleet-watchdog.

Alert states to watch for:

Healthy state: calendar_ok (LastExitStatus=0, plist loaded)

MCTitleStatusNotes
#10286Fix LightRAG ingest LaunchAgents — drain 312 outbox + add watchdogDONE (PARTIAL verify)This fix. Delivered by Kelsey Hightower. Proveo: 3 PASS, 1 PARTIAL, 1 FAIL.
#10298CF Access service token 302 root cause investigationOPEN (priority: M)Why does https://lightrag.alai.no return 302 for local host? Should resolve the need for the localhost bypass.
Infrastructure

Runbook: LightRAG ingest LaunchAgent fix (MC #10286)

Overview

This runbook documents the investigation and fix applied to three LightRAG-related LaunchAgents on the ALAI Mac Studio host in MC #10286. The fix was validated by Proveo (Angie Jones) with a PARTIAL verdict: 3 PASS, 1 PARTIAL (AC3), 1 FAIL (AC4 — same-day unverifiable). CF Access root cause is tracked separately in MC #10298.

1. Symptom — How to Detect This Failure

These signals indicate the com.alai.lightrag-outbox-ingest LaunchAgent is failing silently:

2. Root Cause

The primary failure was in com.alai.lightrag-outbox-ingest:

Workaround applied: Changed LIGHTRAG_URL to http://localhost:9621 in the plist. The CF Access token 302 root cause (why the local host receives a redirect instead of being authorized) is tracked in MC #10298 (priority: M).

The other two daemons were not functionally broken:

3. Fix Procedure

Preconditions: You have shell access to the Mac Studio host. LightRAG is running locally on port 9621.

Step 1: Verify current plist URL

grep -A1 "LIGHTRAG_URL" ~/Library/LaunchAgents/com.alai.lightrag-outbox-ingest.plist

If the value is https://lightrag.alai.no, proceed. If already http://localhost:9621, skip to Step 4.

Step 2: Edit the plist

nano ~/Library/LaunchAgents/com.alai.lightrag-outbox-ingest.plist

Change the LIGHTRAG_URL string value from https://lightrag.alai.no to http://localhost:9621. The correct plist line:

<key>LIGHTRAG_URL</key><string>http://localhost:9621</string>

Step 3: Unload all 3 lightrag plists

launchctl unload ~/Library/LaunchAgents/com.alai.lightrag-outbox-ingest.plist
launchctl unload ~/Library/LaunchAgents/com.alai.lightrag-backup.plist
launchctl unload ~/Library/LaunchAgents/com.john.lightrag-monitor.plist

Step 4: Reload all 3 lightrag plists

launchctl load -w ~/Library/LaunchAgents/com.alai.lightrag-outbox-ingest.plist
launchctl load -w ~/Library/LaunchAgents/com.alai.lightrag-backup.plist
launchctl load -w ~/Library/LaunchAgents/com.john.lightrag-monitor.plist

Step 5: Drain the outbox manually (if backlog exists)

node ~/system/tools/lightrag-outbox-ingest.js

The script is idempotent — it uses outbox-ingest.sqlite with correlation_id as PRIMARY KEY dedup gate. Running it multiple times is safe. Expected output when backlog is cleared: processed: 0, skipped: N, failed: 0.

Step 6: Kickstart the ingest daemon to verify immediate fire

launchctl kickstart -k gui/$(id -u)/com.alai.lightrag-outbox-ingest

Check the log immediately after:

tail -20 ~/system/logs/lightrag-outbox-ingest.log

Expected: A [ingest] DONE line with exit success.

Step 7: Confirm watchdog detects healthy state

bash ~/bin/daemon-fleet-watchdog.sh 2>&1 | grep lightrag

Expected: All 3 labels in calendar_ok state. No calendar_err_* or not_loaded transitions.

4. Verification Commands

# 1. All 3 plists loaded with LastExitStatus=0
launchctl list | grep lightrag

# 2. Checkpoint DB row count (should match mc-task-outcomes.jsonl line count)
sqlite3 ~/system/state/outbox-ingest.sqlite "SELECT count(*) FROM processed"

# 3. Most recent ingest timestamp
sqlite3 ~/system/state/outbox-ingest.sqlite "SELECT MAX(ingested_at) FROM processed"

# 4. LightRAG pipeline health
curl http://localhost:9621/documents/pipeline_status

# 5. LightRAG document total count
curl http://localhost:9621/documents | jq .total

# 6. Outbox log last run summary
grep "DONE" ~/system/logs/lightrag-outbox-ingest.log | tail -5

# 7. Watchdog recent transitions for lightrag
grep lightrag ~/system/logs/daemon-fleet-watchdog.log | tail -20

5. Known Limitations

6. Watchdog Coverage

The daemon-fleet-watchdog at ~/bin/daemon-fleet-watchdog.sh covers all 3 LightRAG plists via its glob at line 39:

for plist in "$HOME"/Library/LaunchAgents/com.{alai,john}.*.plist

This glob automatically includes any new LightRAG LaunchAgents matching the pattern without code changes. The watchdog runs every 15 minutes via com.alai.daemon-fleet-watchdog.

Alert states to watch for:

Healthy state: calendar_ok (LastExitStatus=0, plist loaded)

MCTitleStatusNotes
#10286 Fix LightRAG ingest LaunchAgents — drain 312 outbox + add watchdog DONE (PARTIAL verify) This fix. Delivered by Kelsey Hightower. Proveo: 3 PASS, 1 PARTIAL, 1 FAIL.
#10298 CF Access service token 302 root cause investigation OPEN (priority: M) Why does https://lightrag.alai.no return 302 for local host? Resolves the need for the localhost bypass.
Infrastructure

Runbook: LightRAG ingest LaunchAgent fix (MC #10286)

Overview

This runbook documents the investigation and fix applied to three LightRAG-related LaunchAgents on the ALAI Mac Studio host in MC #10286. The fix was validated by Proveo (Angie Jones) with a PARTIAL verdict: 3 PASS, 1 PARTIAL (AC3), 1 FAIL (AC4 — same-day unverifiable). CF Access root cause is tracked separately in MC #10298.

1. Symptom — How to Detect This Failure

These signals indicate the com.alai.lightrag-outbox-ingest LaunchAgent is failing silently:

2. Root Cause

The primary failure was in com.alai.lightrag-outbox-ingest:

Workaround applied: Changed LIGHTRAG_URL to http://localhost:9621 in the plist. The CF Access token 302 root cause (why the local host receives a redirect instead of being authorized) is tracked in MC #10298 (priority: M).

The other two daemons were not functionally broken:

3. Fix Procedure

Preconditions: You have shell access to the Mac Studio host. LightRAG is running locally on port 9621.

Step 1: Verify current plist URL

grep -A1 "LIGHTRAG_URL" ~/Library/LaunchAgents/com.alai.lightrag-outbox-ingest.plist

If the value is https://lightrag.alai.no, proceed. If already http://localhost:9621, skip to Step 4.

Step 2: Edit the plist

nano ~/Library/LaunchAgents/com.alai.lightrag-outbox-ingest.plist

Change the LIGHTRAG_URL string value from https://lightrag.alai.no to http://localhost:9621. The correct plist line:

<key>LIGHTRAG_URL</key><string>http://localhost:9621</string>

Step 3: Unload all 3 lightrag plists

launchctl unload ~/Library/LaunchAgents/com.alai.lightrag-outbox-ingest.plist
launchctl unload ~/Library/LaunchAgents/com.alai.lightrag-backup.plist
launchctl unload ~/Library/LaunchAgents/com.john.lightrag-monitor.plist

Step 4: Reload all 3 lightrag plists

launchctl load -w ~/Library/LaunchAgents/com.alai.lightrag-outbox-ingest.plist
launchctl load -w ~/Library/LaunchAgents/com.alai.lightrag-backup.plist
launchctl load -w ~/Library/LaunchAgents/com.john.lightrag-monitor.plist

Step 5: Drain the outbox manually (if backlog exists)

node ~/system/tools/lightrag-outbox-ingest.js

The script is idempotent — it uses outbox-ingest.sqlite with correlation_id as PRIMARY KEY dedup gate. Running it multiple times is safe. Expected output when backlog is cleared: processed: 0, skipped: N, failed: 0.

Step 6: Kickstart the ingest daemon to verify immediate fire

launchctl kickstart -k gui/$(id -u)/com.alai.lightrag-outbox-ingest

Check the log immediately after:

tail -20 ~/system/logs/lightrag-outbox-ingest.log

Expected: A [ingest] DONE line with exit success.

Step 7: Confirm watchdog detects healthy state

bash ~/bin/daemon-fleet-watchdog.sh 2>&1 | grep lightrag

Expected: All 3 labels in calendar_ok state. No calendar_err_* or not_loaded transitions.

4. Verification Commands

# 1. All 3 plists loaded with LastExitStatus=0
launchctl list | grep lightrag

# 2. Checkpoint DB row count (should match mc-task-outcomes.jsonl line count)
sqlite3 ~/system/state/outbox-ingest.sqlite "SELECT count(*) FROM processed"

# 3. Most recent ingest timestamp
sqlite3 ~/system/state/outbox-ingest.sqlite "SELECT MAX(ingested_at) FROM processed"

# 4. LightRAG pipeline health
curl http://localhost:9621/documents/pipeline_status

# 5. LightRAG document total count
curl http://localhost:9621/documents | jq .total

# 6. Outbox log last run summary
grep "DONE" ~/system/logs/lightrag-outbox-ingest.log | tail -5

# 7. Watchdog recent transitions for lightrag
grep lightrag ~/system/logs/daemon-fleet-watchdog.log | tail -20

5. Known Limitations

6. Watchdog Coverage

The daemon-fleet-watchdog at ~/bin/daemon-fleet-watchdog.sh covers all 3 LightRAG plists via its glob at line 39:

for plist in "$HOME"/Library/LaunchAgents/com.{alai,john}.*.plist

This glob automatically includes any new LightRAG LaunchAgents matching the pattern without code changes. The watchdog runs every 15 minutes via com.alai.daemon-fleet-watchdog.

Alert states to watch for:

Healthy state: calendar_ok (LastExitStatus=0, plist loaded)

MCTitleStatusNotes
#10286 Fix LightRAG ingest LaunchAgents — drain 312 outbox + add watchdog DONE (PARTIAL verify) This fix. Delivered by Kelsey Hightower. Proveo: 3 PASS, 1 PARTIAL, 1 FAIL.
#10298 CF Access service token 302 root cause investigation OPEN (priority: M) Why does https://lightrag.alai.no return 302 for local host? Resolves the need for the localhost bypass.

How We Work — Project Lifecycle

ALAI Canonical Lifecycle Path

Author: Petter Graff | Date: 2026-04-15

Status: DRAFT — awaiting Angie Jones validation

Design Basis

This document was produced after reading four Phase 1 audit files:

And three reference documents:

Every tool/command referenced below is verified LIVE in the tools-inventory unless explicitly marked MISSING or BROKEN. No tool is referenced from memory.

Immediate Fixes Required (before any lifecycle works)

These two fixes are prerequisites. Nothing else in this document functions without them.

Fix 1: onboard-client.js — wrong scaffold path

File: ~/system/tools/onboard-client.js line 27 Current: const SCAFFOLD = path.join(__dirname, '..', 'template', 'scaffold.sh'); Fix to: const SCAFFOLD = path.join(__dirname, '..', 'templates', 'scaffold', 'scaffold.sh'); Impact: Step 1 of the Saga fails on every run. The entire client onboarding pipeline is blocked.

Fix 2: build-project.js — same wrong scaffold path

File: ~/system/tools/build-project.js line 25 Current: References ~/system/template/ (directory does not exist) Fix to: References ~/system/templates/scaffold/ Impact: Project scaffolding for all internal products and clients fails at first step.

These are two-line changes. They unblock the entire automated pipeline. Do them first, before any lifecycle work begins.

Lifecycle 1: New Company (ALAI Subsidiary)

Examples: FlowForge, Vizu, Proveo, AgentForge.

Step Action Tool/Command BookStack Entry DOD Evidence
1 CEO approves company creation and names it MC task created manually — no automation exists — MC task in OPEN state, CEO explicitly confirmed in Slack or in writing
2 Register company identity in specialist mapping Edit ~/system/agents/specialist-mapping.json manually — grep "<company-name>" ~/system/agents/specialist-mapping.json returns entry
3 Set active company context bash ~/system/tools/active-company.sh set "<company-name>" — bash ~/system/tools/active-company.sh get returns correct name
4 Create company blueprint YAML MISSING — no tool creates this. Create manually at ~/companies/<company-name>/blueprints/<company-name>.yaml. Model on existing CodeCraft/AgentForge/Securion YAML. — node ~/system/tools/blueprint-registry.js list includes new company
5 Scaffold company worktree node ~/system/tools/worktree-company.js create "<company-name>" — git worktree list shows new worktree at ~/companies/<company-name>
6 Assign founding agent identities Edit ~/system/agents/specialist-mapping.json to add agents under company key — node ~/system/tools/agent-manager.js list --company "<company-name>" returns agents
7 Create Slack channel for company node ~/system/tools/slack.js send general "New company <name> created — channel #<name>" then create channel manually — Channel visible in alai-talk.slack.com
8 Sync company knowledge to BookStack node ~/system/tools/bookstack-sync.js sync then manually create "Companies > <CompanyName>" page with mission, agents, routing rules BookStack: Companies > [CompanyName] — mission, agent roster, routing table curl https://docs.alai.no + manual verify page exists
9 Create MC master task for company node ~/system/tools/mc.js add "<CompanyName>: Operational" --priority M --owner john — node ~/system/tools/mc.js show <id> returns task in OPEN state
10 Archive this lifecycle run node ~/system/tools/session-archiver.js save --tag "company-creation-<name>" BookStack: update company page with creation date and MC task ID Session archived, BookStack page updated

Notes:

Lifecycle 2: New Product (Internal)

Examples: Drop, Bilko, Tok, Lobby.

Step Action Tool/Command BookStack Entry DOD Evidence
1 CEO approves product concept node ~/system/tools/mc.js add "<ProductName>: Product Creation" --priority H --owner john — MC task exists, CEO confirmed GO in writing
2 Write BUILD-BLUEPRINT.md for product MISSING as automated step — write manually at ~/ALAI/products/<product>/BUILD-BLUEPRINT.md. Use ALAI-UNIVERSAL-BLUEPRINT.md as template. — File exists: ls ~/ALAI/products/<product>/BUILD-BLUEPRINT.md
3 Scaffold project structure node ~/system/tools/build-project.js scaffold "<ProductName>" --type internal (requires Fix 2 above to work) — ls ~/ALAI/products/<product>/ shows scaffold dirs: src/, docs/, tests/
4 Register product in PLC state cp ~/system/specs/plc-drop-state.json ~/system/specs/plc-<product>-state.json then edit to set product name, phase=1 — cat ~/system/specs/plc-<product>-state.json shows phase: 1
5 Run blueprint compliance baseline node ~/system/tools/blueprint-runner.js run --company alai --blueprint <product>.yaml (requires product YAML first) — node ~/system/tools/blueprint-registry.js show <product> returns BCS score
6 Create product repo and protect main branch Manual: create GitHub repo, enable branch protection, add PR template — git remote -v from product dir returns GitHub URL; branch protection verifiable via GitHub API
7 Assign specialist agents by domain Edit BUILD-BLUEPRINT.md routing table: which CodeCraft agent for backend, which Vizu agent for frontend, etc. — Each domain in BUILD-BLUEPRINT.md has a named agent assigned
8 Sync to BookStack node ~/system/tools/bookstack-sync.js sync then manually create "Products > <ProductName>" page BookStack: Products > [ProductName] — purpose, tech stack, current phase, agent assignments Page exists at docs.alai.no
9 Create Sprint 1 MC tasks node ~/system/tools/mc.js add "<ProductName>: Sprint 1" --priority H --route backend (repeat per domain) — node ~/system/tools/mc.js list shows Sprint 1 tasks assigned
10 Declare product ACTIVE in PLC Edit plc-<product>-state.json to set phase=2, status=active BookStack: update product page with phase and MC master task ID node ~/system/tools/mc.js show <master-task-id> shows product task chain

Notes:

Lifecycle 3: New Client (External)

Examples: Braive, LumisCare, Nordic Wizard.

Step Action Tool/Command BookStack Entry DOD Evidence
1 Record first contact NODE_PATH=~/system/node_modules node ~/system/tools/contacts.js add "<Name>" "<email>" --company "<Firm>" --type client --notes "<description>" then node ~/system/tools/sales-pipeline.js add "<Firm>" "<email>" "<source>" "<description>" — node ~/system/tools/contacts.js search "<name>" returns entry; node ~/system/tools/sales-pipeline.js list shows lead
2 Run discovery call, write brief Manual: gather problem, budget, timeline, platforms, integrations. Write ~/ALAI/clients/<CLIENT>/intake/discovery-notes.md and project-brief.md — Both files exist on disk: ls ~/ALAI/clients/<CLIENT>/intake/
3 NDA signed NODE_PATH=~/system/node_modules node ~/system/tools/docusign.js create "<CLIENT>" nda --field CLIENT_NAME="<name>" --field CLIENT_EMAIL="<email>" then /send-for-signing skill. TEST ON post@alai.no FIRST. — Signed PDF at ~/ALAI/clients/<CLIENT>/legal/nda-signed.pdf (DocuSeal confirmation email received)
4 Proposal drafted and CEO-approved NODE_PATH=~/system/node_modules node ~/system/tools/proposal-gen.js create "<CLIENT>" then present to Alem for GO. ZAKON: NEVER send pricing without CEO explicit GO. — Alem has said "GO" or "SEND" explicitly. No other gate passes.
5 Contract signed and first payment received node ~/system/tools/docusign.js create "<CLIENT>" contract ... then /send-for-signing. Then node ~/system/tools/invoice-generator.js create "<CLIENT>" <amount> NOK "Project kickoff" — Signed contract PDF exists; Fiken shows payment received: node ~/system/tools/fiken.js invoices list --client "<CLIENT>"
6 Project scaffolded NODE_PATH=~/system/node_modules node ~/system/tools/onboard-client.js new "<slug>" "<email>" "<source>" "<value>" "<description>" (requires Fix 1 above) — ls ~/projects/<slug>/ returns directory with scaffold structure
7 Sales pipeline advanced to WON node ~/system/tools/sales-pipeline.js advance <lead-id> "Contract signed, project started" --approved — node ~/system/tools/sales-pipeline.js show <lead-id> returns stage: WON
8 Client page created in BookStack node ~/system/tools/bookstack-sync.js sync then manually create "Clients > <ClientName>" page with: contact, brief, contract date, assigned agents, project slug BookStack: Clients > [ClientName] — contact info, project brief summary, signed date, assigned team, sprint link Page exists, contains all required fields
9 Sprint 1 planned, agents assigned node ~/system/tools/mc.js add "<CLIENT>: Sprint 1" --priority H --route backend (repeat per domain). Assign to appropriate specialist agents per CLAUDE.md routing table. — All Sprint 1 tasks in MC with assigned agents, priority H
10 Client status update sent node ~/system/tools/client-status-update.js send "<CLIENT>" "Project kickoff complete. Sprint 1 underway." BookStack: update client page with Sprint 1 start date Client receives written confirmation; MC task for sprint 1 is in STARTED state

Notes:

Lifecycle 4: New Project — Day-to-Day (Start to Deploy to Done)

This covers the standard sprint execution loop. Applies to all active products and client projects.

Step Action Tool/Command BookStack Entry DOD Evidence
1 Task received, classified, routed John classifies (build/research/infra/design/QA/finance). Routes to specialist agent per CLAUDE.md routing table. node ~/system/tools/mc.js add "<task>" --priority <H/M/L> --route <backend/frontend/infra/qa> — node ~/system/tools/mc.js show <id> returns correct owner and priority
2 Agent starts task, loads context node ~/system/tools/mc.js start <id>. Agent reads BUILD-BLUEPRINT.md. Agent runs node ~/system/tools/context-loader.js <id> for task context bundle. — node ~/system/tools/mc.js show <id> returns status: STARTED, start_time set
3 Build with blueprint compliance Agent builds. Runs node ~/system/tools/preflight-check.js before coding. Follows master-blueprint.md 13-domain requirements (MUST tier is non-negotiable). — node ~/system/tools/preflight-check.js exits 0
4 Test — 5 clean iterations minimum node ~/system/tools/qa-19.js run <project> --iterations 5. Requires: 15/19 minimum score, 17/19 for HIGH priority. All 5 test levels must exist (unit/integration/e2e/regression/performance). — node ~/system/tools/qa-19.js show <project> returns score >= 15/19 (or 17/19 for H). ls tests/logs/iteration-*.log shows 5 files.
5 Pre-deploy gate bash ~/system/tools/gate-pre-deploy.sh <project>. Also runs node ~/system/tools/deploy-gate.js. — Both gates exit 0. No advisory-only pass accepted.
6 Deploy node ~/system/tools/deploy-manager.js deploy <project> --env staging. After staging verification: node ~/system/tools/deploy-manager.js deploy <project> --env production. Uses Vercel/Railway/Fly.io per product config. — node ~/system/tools/deploy-verify.sh <project> <env> returns PASS. Playwright browser test confirms real UI renders (not just curl 200).
7 Post-deploy smoke test node ~/system/tools/smoke-test.js run <project> --env production — node ~/system/tools/smoke-test.js show <project> returns all checks PASS
8 Agent marks READY — not DONE node ~/system/tools/mc.js ready <id> "<outcome summary>". Agent CANNOT self-declare DONE. Only Proveo QA can advance to DONE. — node ~/system/tools/mc.js show <id> returns status: READY_FOR_REVIEW
9 Proveo validates (Angie Jones) Proveo runs node ~/system/tools/qa-19.js check <id> against ungameable-testing-methodology.md standard. Checks: real browser test was run, 5 clean iterations logged, blueprint compliance score returned. — node ~/system/tools/mc.js show <id> status changed to DONE only by Proveo agent, never by builder
10 Sync to BookStack and close node ~/system/tools/bookstack-sync.js sync. Manually update project page with: what shipped, deploy URL, version, date, any known issues. Then node ~/system/tools/mc.js done <id> "<final outcome>" BookStack: Project page updated with shipped feature, deploy URL, version tag, completion date node ~/system/tools/mc.js show <id> returns status: DONE. BookStack page updated. Session archived: node ~/system/tools/session-archiver.js save --tag "task-<id>"

Notes:

What to Build vs What Already Exists

Already Exists and Works (after the two path fixes)

Tool Status Notes
onboard-client.js LIVE-BROKEN -> LIVE after Fix 1 Saga pattern, 8 steps, well-built
build-project.js LIVE-BROKEN -> LIVE after Fix 2 Scaffold + spec + MC task
sales-pipeline.js LIVE-FUNCTIONAL Lead lifecycle, WON enforcement
contacts.js LIVE-FUNCTIONAL Contact management
invoice-generator.js LIVE-FUNCTIONAL MVA auto-applied
docusign.js / send-signing-email.js LIVE-FUNCTIONAL NDA + contract signing
mc.js LIVE-FUNCTIONAL Task lifecycle, ready/done separation
qa-19.js LIVE-FUNCTIONAL QA gate, 15/19 and 17/19 thresholds
gate-pre-deploy.sh / deploy-gate.js LIVE-FUNCTIONAL Pre-deploy enforcement
deploy-manager.js / deploy-verify.sh LIVE-FUNCTIONAL Deploy + verification
smoke-test.js LIVE-FUNCTIONAL Post-deploy smoke
bookstack-sync.js LIVE-FUNCTIONAL Works when called; never auto-triggered
blueprint-registry.js LIVE-FUNCTIONAL Works when called; never auto-triggered
blueprint-runner.js LIVE-BROKEN Ran once and failed. Needs diagnostic before relying on it
session-archiver.js LIVE-FUNCTIONAL Session archival
slack.js LIVE-FUNCTIONAL Slack messaging
worktree-company.js LIVE-FUNCTIONAL Git worktree per company
context-loader.js LIVE-FUNCTIONAL Task context bundle for agents
preflight-check.js LIVE-FUNCTIONAL Pre-build checks
agent-manager.js LIVE-FUNCTIONAL Agent lifecycle

Needs to Be Built

Missing Capability Priority Notes
BookStack auto-trigger on lifecycle events HIGH Currently zero tools call bookstack-sync.js automatically. Every lifecycle step that writes to BookStack is currently manual. A post-hook or step in onboard-client.js and build-project.js is needed.
onboard-client.js Step 9: BookStack page creation HIGH The Saga ends at event logging. A Step 9 that calls bookstack-sync.js to create the client page would close this gap without redesigning the tool.
Company creation automation MEDIUM No tool creates a company end-to-end. Lifecycle 1 is currently almost entirely manual. A new-company.js tool (modeled on onboard-client.js Saga pattern) would unify this.
autocoder.js MEDIUM ACTIVE spec (alai-autocoder.md) targets this file. It does not exist. LumisCare's build plan (582 features) is explicitly blocked on it. Build AutoCoder before assigning LumisCare to the AI Services Pivot delivery pipeline.
blueprint-runner.js diagnosis and fix MEDIUM Ran once, failed. Until it runs successfully, the automated blueprint compliance gate (master-blueprint.md enforcement) is non-functional. Diagnose the 2026-03-09 failure before the next product launch.
Product creation automation LOW Lifecycle 2 is mostly manual steps. A new-product.js that mirrors onboard-client.js would reduce friction. Not blocking — manual steps work.

Known Contradictions to Resolve Before Phase 2 Deletion Runs

Per Angie Jones cross-validation:

Enforcement Rule: Builder Cannot Say Done

This is not optional. It is encoded in the separation of mc.js ready (builder calls) vs mc.js done (Proveo calls only).

The pattern from six abandoned specs (john-orchestrator-fix.md, auto-verify-system.md, root-cause-fix-7307-plan.md, enforcement-upgrade-plan.md, deterministic-enforcement-plan.md — all ABANDONED) is that the enforcement was designed but never made mechanical. global-dod-system-plan.md is the seventh iteration and the only currently ACTIVE spec.

The canonical path above makes this mechanical: Step 8 in Lifecycle 4 is mc.js ready. Step 9 is Proveo's mc.js done. The builder cannot execute Step 9. If this is not enforced at the tooling layer, the canonical path will fail the same way all six previous specs failed.

Author: Petter Graff | CodeCraft | 2026-04-15 Validation required: Angie Jones (Proveo) before any lifecycle is declared operational Immediate unblocking: Fix 1 + Fix 2 (two-line path corrections) before any other work

ZAKON PLAN Linter

Runbook: ZAKON PLAN Linter

Owner: Proveo
File: ~/system/tools/zakon-plan-lint.sh
Version: 1.0.0
Last Updated: 2026-04-16

Purpose

Enforce ZAKON PLAN compliance: every plan MUST include validation task (Proveo/Angie) and documentation task (Skillforge/BookStack). This is a non-negotiable Hard Constraint from ~/.claude/CLAUDE.md.

Problem solved: Plans were frequently shipped without these mandatory tasks. This linter makes compliance technical, not voluntary.

How It Works

The linter scans a plan file (Markdown) for:

  1. Validation task indicators:

    • Owner field containing: Proveo, Angie, angie-jones, V1 (validator role)
    • Task description containing: validation, end-to-end, evidence, Proveo sign-off

  2. Documentation task indicators:

    • Owner field containing: Skillforge, D1 (docs role)
    • Task description containing: documentation, BookStack, runbook

Detection method: Pattern matching with case-insensitive regex.

Exit codes:

Usage

Command-Line

bash ~/system/tools/zakon-plan-lint.sh <path-to-plan.md>

Examples:

# Check a single plan
bash ~/system/tools/zakon-plan-lint.sh ~/system/specs/system-evolution-plan.md
# Output: ✅ ZAKON PLAN COMPLIANT

# Check a non-compliant plan
bash ~/system/tools/zakon-plan-lint.sh ~/system/specs/old-plan.md
# Output: ❌ MISSING validation task
# Exit code: 1

# Use in CI/validation script
if bash ~/system/tools/zakon-plan-lint.sh ~/system/specs/my-plan.md; then
  echo "Plan approved"
else
  echo "Plan rejected — add validation + docs tasks"
  exit 1
fi

Pre-Commit Hook (Recommended)

Automatically check plans before committing to git:

Setup:

# Create pre-commit hook
cat > ~/ALAI/.git/hooks/pre-commit << 'EOF'
#!/bin/bash
# ZAKON PLAN linter — runs on all staged *-plan.md files

STAGED_PLANS=$(git diff --cached --name-only --diff-filter=ACM | grep 'specs/.*-plan\.md$')

if [ -z "$STAGED_PLANS" ]; then
  exit 0  # No plans staged, skip check
fi

FAILED=0
for PLAN in $STAGED_PLANS; do
  if [ -f "$PLAN" ]; then
    if ! bash ~/system/tools/zakon-plan-lint.sh "$PLAN"; then
      echo "❌ $PLAN is not ZAKON PLAN compliant"
      FAILED=1
    fi
  fi
done

if [ $FAILED -eq 1 ]; then
  echo ""
  echo "Fix: Add validation task (Proveo/Angie) and documentation task (Skillforge) to your plan."
  exit 1
fi
EOF

chmod +x ~/ALAI/.git/hooks/pre-commit

Now every commit with a plan file will be checked automatically.

Output Format

Compliant Plan

✅ ZAKON PLAN COMPLIANT: /Users/makinja/system/specs/system-evolution-plan.md
  [✓] Validation task found (owner: V1, task: end-to-end evidence run)
  [✓] Documentation task found (owner: D1, task: BookStack runbooks)

Non-Compliant Plan

❌ ZAKON PLAN VIOLATION: /Users/makinja/system/specs/my-plan.md
  [✗] MISSING: Validation task (must include Proveo/Angie owner)
  [✗] MISSING: Documentation task (must include Skillforge owner)

Required:
  • Add task with owner: Proveo / Angie / V1 (validation role)
  • Add task with owner: Skillforge / D1 (docs role)

What Gets Detected

Validation Task Keywords

Example (PASS):

**Task 14 (VALIDATION — MANDATORY):** End-to-end evidence run
- Owner: V1 (Angie Jones / Proveo)
- Acceptance:
  - [ ] Synthetic task validated with real evidence
  - [ ] Proveo sign-off in MC task

Documentation Task Keywords

Example (PASS):

**Task 15 (DOCS — MANDATORY):** BookStack runbooks
- Owner: D1 (Skillforge)
- Acceptance:
  - [ ] BookStack page created
  - [ ] Runbooks published

When to Use

Always Run For:

Integration Points:

  1. mc.js add with plan: mc.js add --plan-ref <path> should auto-lint
  2. Pre-commit hook: Git commit of plan files
  3. Weekly regression suite: system-regression.sh scans all plans (max 10)
  4. Manual review: John reviewing plan before CEO approval

Bypass Procedure (EMERGENCY ONLY)

You should NEVER bypass this linter. If a plan fails, fix the plan.

However, in true emergency (production incident, CEO direct override):

  1. Skip pre-commit hook:

    git commit --no-verify -m "Emergency deploy"

  2. Forced MC task creation:

    # There is NO bypass flag — linter is read-only check
# Fix the plan or discuss with Petter Graff

  3. Rationale: Bypassing means shipping a plan that will fail at Phase 4. Builder will finish → no validator → no docs → system regresses. CEO Hard Constraint #2: "No claim without evidence."

Troubleshooting

Linter Says Missing But Task Exists

Symptom: You added validation task, but linter still rejects.

Fix: Ensure keywords are in plaintext (not code blocks) and match patterns:

# WRONG (in code block)
`Owner: Angie`

# CORRECT (plain Markdown)
Owner: Angie Jones (Proveo)

Linter Detects Wrong Section as Task

Symptom: Linter passes but you didn't add tasks — it detected something in background context.

Fix: This is edge case but acceptable. If pattern appears in "Related Work" section, manual review will catch it. Linter is first gate, not only gate.

What If Plan is Too Small for Validation?

Answer: Every plan needs validation, even tiny ones. Scale validation to match:

Documentation also scales:

Hard rule: Size doesn't exempt. Every plan enriches system knowledge.

Integration with Other Gates

flowchart TD
    A[Plan Draft] -->|ZAKON linter| B{Compliant?}
    B -->|No| C[Reject: Fix Plan]
    B -->|Yes| D[Git Commit]
    D -->|Pre-commit hook| E[Linter re-runs]
    E -->|Pass| F[Plan in Repo]
    F --> G[Builder Starts]
    G --> H[Task Done]
    H -->|Proveo Gate| I{Evidence?}
    I -->|No| J[Reject: Need Validation]
    I -->|Yes| K[Task Complete]
    K --> L[Skillforge Docs]
    L --> M[Plan Fully Compliant]
    
    style B fill:#fff3cd
    style I fill:#fff3cd
    style M fill:#e1f5e1

Three gates work together:

  1. ZAKON linter: Checks plan structure (this runbook)
  2. Proveo gate: Checks task evidence before mc.js done (see ~/system/docs/runbooks/mc-done-proveo-gate.md)
  3. Blueprint liveness: Checks blueprint mtime (see ~/system/docs/runbooks/blueprint-liveness.md)

All three enforce Hard Constraints from ~/.claude/CLAUDE.md.

Exit Codes Reference

Code Meaning Action
0 Plan compliant Proceed
1 Missing validation or docs task Fix plan
2 File not found / not readable Check path

Maintenance

Adding New Keywords

If agents start using new terms (e.g., "QA task" instead of "validation task"), update patterns:

File: ~/system/tools/zakon-plan-lint.sh

# Validation keywords (case-insensitive grep)
VALIDATION_PATTERNS="Proveo|Angie|angie-jones|V1|validator|validation|QA"

# Documentation keywords
DOCS_PATTERNS="Skillforge|D1|documentation|BookStack|runbook|docs"

Test after change:

bash ~/system/tools/zakon-plan-lint.sh ~/system/specs/system-evolution-plan.md && echo PASS

False Positives Log

Track patterns that incorrectly pass/fail:

Location: ~/system/logs/zakon-linter-audit.jsonl

{"timestamp": "2026-04-20T10:30:00Z", "plan": "my-plan.md", "issue": "Detected 'validation' in unrelated section", "severity": "low"}

Review monthly. If >5 false positives/month, refine regex.

Examples

Example 1: Minimal Compliant Plan

# Plan: Add New Feature

## Tasks

**Task 1:** Implement feature X
- Owner: CodeCraft
- Acceptance: Feature works

**Task 2 (VALIDATION):** E2E test
- Owner: Angie Jones
- Acceptance: Evidence bundle in ~/system/evidence/

**Task 3 (DOCS):** BookStack page
- Owner: Skillforge
- Acceptance: Page published at docs.alai.no

Linter result: ✅ PASS

Example 2: Non-Compliant Plan (Missing Docs)

# Plan: Quick Fix

## Tasks

**Task 1:** Fix bug
- Owner: CodeCraft

**Task 2:** Test fix
- Owner: Proveo

Linter result: ❌ FAIL — Missing documentation task

Example 3: Tricky But Valid

# Plan: System Upgrade

## Phase 1
Tasks 1-10 by various builders...

## Phase 2: MANDATORY Validation + Docs

**Task 11 (VALIDATION):** End-to-end validation
- Owner: V1 (Proveo)

**Task 12 (DOCS):** Update runbooks
- Owner: D1 (Skillforge)

Linter result: ✅ PASS (tasks can be anywhere in file, not just at end)

Changelog

Date Version Change
2026-04-16 1.0.0 Initial implementation (MC #8020 Task 8)

Questions? Contact Petter Graff (team lead) or read ~/system/rules/john-operating-system.md section on ZAKON PLAN.

LightRAG Default-On in discover.js

Runbook: LightRAG Default-On in discover.js

Owner: AgentForge
File: ~/system/tools/discover.js
Version: 2.0.0
Last Updated: 2026-04-16

Purpose

Make LightRAG graph retrieval the default for all discover.js queries. Before this change, LightRAG was opt-in (--lightrag flag). After, it's opt-out (--no-lightrag flag).

Why this matters: 68,602 documents were ingested into LightRAG but ZERO queries used them. Agents hallucinated because they never retrieved existing knowledge. This change closes the retrieval gap.

What Changed

Before (Opt-In)

const useLightRAG = flags.lightrag || false;

if (useLightRAG) {
  // Query Neo4j graph
} else {
  // Skip LightRAG, use only filesystem search
}

Usage:

node ~/system/tools/discover.js "query"             # No LightRAG
node ~/system/tools/discover.js --lightrag "query"  # With LightRAG

Problem: Agents never used --lightrag flag. Default workflow bypassed graph.

After (Opt-Out)

const useLightRAG = !flags['no-lightrag'];  // Default TRUE

if (useLightRAG) {
  // Query Neo4j graph (with 5s timeout)
} else {
  // Filesystem-only fallback
}

Usage:

node ~/system/tools/discover.js "query"                # LightRAG enabled
node ~/system/tools/discover.js --no-lightrag "query"  # LightRAG disabled

Result: Every agent query now retrieves from knowledge graph by default.

How It Works

Query Flow

flowchart TD
    A[discover.js called] -->|Default| B{--no-lightrag flag?}
    B -->|No| C[Query LightRAG]
    B -->|Yes| D[Skip to Filesystem]
    C -->|Timeout 5s| E{Response?}
    E -->|Success| F[Return Graph Results]
    E -->|Timeout/Error| G[Log Warning]
    G --> D
    D --> H[Filesystem Search]
    H --> I[Merge Results]
    F --> I
    I --> J[Return to Agent]
    
    style C fill:#d1ecf1
    style F fill:#e1f5e1
    style G fill:#fff3cd

Key features:

  1. Timeout protection: LightRAG query has 5s timeout
  2. Fallback: If LightRAG fails/times out, filesystem search still runs
  3. Non-blocking: LightRAG unavailability doesn't crash discover.js

When to Use --no-lightrag Flag

Use Cases for Disabling LightRAG

1. LightRAG is down/unhealthy:

# Check container status
docker inspect lightrag | jq -r '.State.Health.Status'
# If "unhealthy" or "starting", use --no-lightrag

node ~/system/tools/discover.js --no-lightrag "query"

2. Debugging filesystem search:

# Compare results with/without graph
node ~/system/tools/discover.js "agent routing" > /tmp/with-graph.txt
node ~/system/tools/discover.js --no-lightrag "agent routing" > /tmp/without-graph.txt
diff /tmp/with-graph.txt /tmp/without-graph.txt

3. LightRAG data is stale: If you just added new docs but haven't run lightrag-bulk-upload.js, graph won't have them:

# Filesystem has latest, graph is stale
node ~/system/tools/discover.js --no-lightrag "new feature X"

4. Performance testing: Measure filesystem-only latency vs. graph+filesystem:

time node ~/system/tools/discover.js --no-lightrag "products" > /dev/null
time node ~/system/tools/discover.js "products" > /dev/null

Verification

Check LightRAG is Being Queried

node ~/system/tools/discover.js "MC task workflow" | grep -A 5 "LightRAG"

Expected output:

=== LightRAG Results ===
Entities: mission-control, workflow, task-lifecycle
Relationships: mc.js -> database -> sqlite
Confidence: 0.87

If you see "LightRAG: skipped" or no section, either:

Check Timeout Behavior

Force a timeout test:

# Stop LightRAG temporarily
docker stop lightrag

# discover.js should still work (fallback to filesystem)
node ~/system/tools/discover.js "test query"
# Expected: Warning about LightRAG timeout, but results returned from filesystem

# Restart
docker start lightrag

Verify Flag Works

# With LightRAG (default)
node ~/system/tools/discover.js "agents" | wc -l

# Without LightRAG
node ~/system/tools/discover.js --no-lightrag "agents" | wc -l

# Second command should have fewer lines (no graph results section)

Performance Characteristics

Latency Budget

Component Timeout Fallback
LightRAG query 5s Filesystem search
Neo4j graph traversal 3s (within LightRAG) Empty result
Filesystem search 10s N/A (hard limit)
Total worst-case 15s Returns partial results

Average response time:

Token Cost Impact

LightRAG results are appended to discover.js output, increasing context size:

Typical increase:

Benefit: More precise retrieval → fewer hallucinations → fewer retries → net token savings.

Measurement (2026-04-16 data):

Troubleshooting

Issue 1: "LightRAG timeout" in Every Query

Symptoms:

[WARN] LightRAG query timed out after 5000ms
Falling back to filesystem search...

Diagnosis:

# Check container
docker ps | grep lightrag
docker logs lightrag --tail 50

# Check Neo4j (LightRAG backend)
docker logs neo4j --tail 50 | grep -i error

# Check load
curl -s http://localhost:9621/documents | jq '.statuses.pending'
# If pending > 50,000 → heavy ingest load causing timeouts

Fix:

  1. Temporary: Use --no-lightrag flag until ingest completes
  2. Long-term: Increase probe timeout (see ~/system/docs/system-evolution-2026-04-16.md Issue #1)

Issue 2: LightRAG Returns Irrelevant Results

Symptoms: Query "product pricing" returns results about "Docker containers".

Diagnosis:

# Check what's indexed
curl -s http://localhost:9621/documents | jq '.statuses | {processed, failed}'

# Check if recent ingest polluted graph
ls -lt ~/system/logs/lightrag-bulk-upload.log | head -1

Fix:

  1. Refine query with more specific terms: discover.js "ALAI product pricing 2026"
  2. Check ~/system/docs/system-evolution-2026-04-16.md section on LightRAG data quality
  3. If graph is corrupted: escalate to Chip Huyen (AgentForge lead)

Issue 3: discover.js Slower After Upgrade

Symptoms: Queries now take 3-5s vs. 1s before.

Expected: This is normal. LightRAG adds 1-2s latency.

Optimization:

# Measure breakdown
node ~/system/tools/discover.js --debug "query" 2>&1 | grep "elapsed"

# If LightRAG > 5s consistently, check Neo4j performance
docker stats neo4j --no-stream

Workaround: For time-sensitive queries, use --no-lightrag flag.

Issue 4: Graph Results Don't Match Filesystem

Symptoms: discover.js "MC tasks" returns filesystem hits but LightRAG says "no results".

Cause: LightRAG ingest lag. Graph is up to 24h behind filesystem.

Check lag:

# Last ingest time
curl -s http://localhost:9621/status | jq .last_ingest_timestamp

# Compare to file mtime
ls -l ~/system/databases/mission-control.db

Fix: Trigger manual ingest:

node ~/system/tools/lightrag-bulk-upload.js ~/system/databases/

Integration with Other Tools

In Agent Chains

Agents calling discover.js automatically get LightRAG:

Chain YAML:

- step: research
  agent: john
  task: "Find all information about Plock product"
  tools:
    - discover.js  # LightRAG enabled by default

No changes needed. Existing chains get graph retrieval.

In Subagents

Subagents spawned by John inherit discover.js behavior:

// Subagent code
const results = await bash(`node ~/system/tools/discover.js "agent routing"`);
// LightRAG results included automatically

In Scripts

Custom scripts can control LightRAG:

#!/bin/bash
# my-script.sh

if [ "$LIGHTRAG_AVAILABLE" = "true" ]; then
  node ~/system/tools/discover.js "query"
else
  node ~/system/tools/discover.js --no-lightrag "query"
fi

Monitoring & Observability

Daily Health Check

# Add to daily briefing
node ~/system/tools/discover.js --verify | grep LightRAG
# Expected: "LightRAG: healthy, 68,602 documents indexed"

Metrics to Track

Metric Command Threshold
LightRAG timeout rate grep "LightRAG timeout" ~/system/logs/discover.log | wc -l < 5/day
Graph hit rate grep "LightRAG Results" ~/system/logs/discover.log | wc -l > 80%
Average latency Parse discover.log for elapsed time < 3s p95

Alert triggers:

Rollback Procedure

If LightRAG default-on causes issues, temporarily revert:

cd ~/system/tools
git log discover.js | head -5  # Find commit before upgrade

# Edit discover.js, line ~87
# Change:
#   const useLightRAG = !flags['no-lightrag'];
# To:
#   const useLightRAG = flags.lightrag || false;

# Test
node ~/system/tools/discover.js "test"  # Should NOT query LightRAG
node ~/system/tools/discover.js --lightrag "test"  # Should query LightRAG

Report rollback to: Chip Huyen (AgentForge) + Petter Graff with error logs.

Future Enhancements

Planned (MC #8050)

  1. Smart timeout: Adjust timeout based on pending ingest queue size
  2. Caching: Cache frequent queries (TTL 5 min)
  3. Federated search: Query multiple graphs (HiveMind + LightRAG + BookStack)

Under Consideration

JWT Bearer Auth (MC #105189, 2026-07-10)

LightRAG server requires JWT Bearer auth in addition to CF Access at the edge. Until 2026-07-10 searchLightRAG sent only CF Access headers, so every LightRAG query returned 401 {"detail":"No credentials provided"} — and the error body was parsed and displayed as a fake "LIGHTRAG (1 match)". LightRAG was silently absent from discovery while --verify said "OK — reachable".

Fixed in three places:

  1. discover.js now awaits getLightRAGBearerHeaders() from the shared ~/system/tools/lightrag-auth-helper.js (MC #103947: Vaultwarden password → POST /login → in-memory-cached JWT) and sends the Authorization header. Helper is fail-open; on failure the server's 401 is surfaced via a new non-200 status check as [error] LightRAG HTTP 401: ... instead of being rendered as a match.
  2. discover.js --verify runs an authenticated query probe and reports OK — authenticated query returned N match(es) / FAIL — <error> / UNREACHABLE. "Reachable" alone is no longer treated as healthy.
  3. lightrag.js query gained --context-only (only_need_context: true) — retrieval-only (~1–3s), skipping server-side LLM answer generation which depends on the FORGE/ollama backend and routinely exceeded the 30s client timeout.

Evidence: ~/system/evidence/105189/lightrag-internal-first-fix-20260710.md (positive, verify, context-only, and adversarial no-credentials runs, all live 2026-07-10).

Note: discover.js memory (the per-prompt PILOT hook path) is local memory-file scoring, not LightRAG — LightRAG participates via the full-search form discover.js "<query>".

Builder Context and Token Accounting (MC #105573, 2026-08-09)

Pi orchestrator now obtains builder knowledge through one authenticated path: pi-orchestrator.js calls rag-context-for-builder.js, which calls lightrag.js and the shared JWT helper. The old duplicate unauthenticated curl query was removed.

Operational behavior:

  1. lightrag-auth-helper.js normalizes a legacy LIGHTRAG_URL ending in /query before calling /login, so the installed LaunchAgent configuration remains compatible.
  2. Only the answer portion of a LightRAG response is eligible for injection. Echoed query text, banners, and footers are excluded from relevance scoring.
  3. L3 context is omitted unless it overlaps the task by at least one meaningful keyword. Override with LIGHTRAG_BUILDER_MIN_KEYWORD_HITS only after measuring retrieval quality.
  4. The route=NULL eligibility protection is unchanged.
  5. Anthropic cache-read and cache-creation token counters are preserved end to end and stored in costs.db; legacy cached_tokens remains a cache-read alias.

Verification commands:

node ~/system/tests/lightrag-token-savings.test.js
env LIGHTRAG_URL=https://lightrag.alai.no/query \
  LIGHTRAG_BUILDER_TIMEOUT_MS=12000 \
  node ~/system/tools/rag-context-for-builder.js \
  "Context Engineering Reform orchestrator token cost" --max-tokens 400
launchctl print gui/$(id -u)/com.john.pi-orchestrator

The production probe on 2026-08-09 returned relevant L3 context in 4.11 seconds. Full evidence, including test output and the distinction between local cache heuristics and provider-billed token usage, is stored in ~/system/evidence/lightrag-token-savings-20260809/verification.md.

The true metered prompt-cache A/B remains pending until a usable Anthropic API key or CLI quota is available. Re-run node ~/system/tools/prompt-cache-ab.js --live; use --cli-live only as a subscription/notional fallback.

Durable writeback queue recovery

The 2026-08-09 completion check also found state/ingest-queue.sqlite malformed. The database was copied to the evidence backup directory, recovered into a new SQLite file, validated with PRAGMA quick_check, and swapped only while no process held it open. Recovery preserved 6,533 processed hashes, two adapter watermarks, and 82 lost-and-found rows. The failed MC #105573 item was then requeued and uploaded in 83 ms; final queue depth was zero. Never overwrite a damaged queue in place: retain the original, recover to a new path, validate, and use an atomic rename.

Changelog

Date Version Change
2026-08-09 2.2.0 Authenticated relevance-gated builder context and cache usage accounting (MC #105573)
2026-07-10 2.1.0 JWT Bearer auth wired via lightrag-auth-helper; authenticated --verify probe; non-200 → error not match; lightrag.js --context-only (MC #105189)
2026-04-16 2.0.0 LightRAG default-on (MC #8020 Task 4)
2026-03-10 1.5.0 Added --lightrag opt-in flag
2026-02-18 1.0.0 Initial discover.js release

Questions? Contact Chip Huyen (AgentForge lead) or check ~/system/docs/system-evolution-2026-04-16.md.

mc.js done — Auto-Writeback to HiveMind + LightRAG Outbox

Runbook: MC Done Auto-Writeback to HiveMind

Owner: AgentForge
File: ~/system/tools/mc.js (done command)
Version: 2.1.0
Last Updated: 2026-05-26

2026-05-26 Reliability Update (MC #102083)

MC completion writeback is now non-blocking on LightRAG availability:

Evidence bundle for this update: ~/system/evidence/102083/wp4-writeback-reliability-report.md. P2P verifier evidence: Company Mesh thread mesh-thr-f759f9d2-a62d-491d-9ecb-677fcfd808fd.

Purpose

Automatically capture task learnings when mc.js done <id> is called and write them to HiveMind + LightRAG pipeline. This closes the learning loop: task completion → knowledge indexing → next agent retrieval → smarter execution.

Before: Task learnings stayed in session logs. Next agent started from zero context.
After: Every completed task enriches the system's institutional memory.

What Gets Captured

When mc.js done <id> runs, it extracts and stores:

Field Source Example
task_id MC task ID 8020
title Task title "System Evo T11: Blueprint liveness gate"
outcome Done command message "Gate implemented. mc.js checks blueprint mtime during done."
owner Task owner "john"
duration Start → done delta "2h 34m"
tags Auto-extracted ["mc", "blueprint", "governance"]
quality_gate Proveo validation "passed" / "bypassed"

Additional context (if available):

Write Destinations

1. HiveMind (Immediate)

Target: ~/system/databases/hivemind.db → intel table

Schema:

INSERT INTO intel (
  category,       -- "briefing"
  content,        -- Structured summary
  source,         -- "mc-done"
  metadata,       -- JSON blob
  created_at      -- ISO timestamp
) VALUES (?, ?, ?, ?, ?);

Write mode: Fire-and-forget async. Non-blocking.

Example entry:

category: "briefing"
content: "Task #8020 (Blueprint liveness gate) completed by john. Outcome: Gate implemented. mc.js checks blueprint mtime during done. Duration: 2h 34m. Quality gate: passed."
source: "mc-done"
metadata: {"task_id":8020,"owner":"john","tags":["mc","blueprint"],"session_id":"731c913c"}
created_at: "2026-04-16T21:12:03Z"

2. Outbox (Deferred Bulk Ingest)

Target: ~/system/logs/mc-task-outcomes.jsonl

Format: JSON Lines (one task per line)

Example:

{"task_id":8020,"title":"Blueprint liveness gate","outcome":"Gate implemented","owner":"john","completed_at":"2026-04-16T21:12:03Z","duration_minutes":154,"tags":["mc","blueprint","governance"],"quality_gate":"passed","session_id":"731c913c","evidence_ref":"/Users/makinja/system/evidence/system-evolution-2026-04-16/"}

Consumption: Bulk-uploaded to LightRAG nightly by lightrag-bulk-upload.js (cron 03:00).

Why two destinations?

Flow Diagram

sequenceDiagram
    participant Agent
    participant mc.js
    participant HiveMind
    participant Outbox
    participant LightRAG
    participant NextAgent

    Agent->>mc.js: done 8020 "outcome text"
    mc.js->>mc.js: Extract task summary
    mc.js->>mc.js: Build intel entry
    
    par Write to HiveMind
        mc.js->>HiveMind: INSERT intel (fire-and-forget)
        HiveMind-->>mc.js: ACK (or log error)
    and Append to Outbox
        mc.js->>Outbox: Append JSONL line
    end
    
    mc.js->>Agent: Task marked done
    
    Note over Outbox,LightRAG: Nightly at 03:00
    LightRAG->>Outbox: Read new JSONL entries
    LightRAG->>LightRAG: Ingest to Neo4j graph
    
    NextAgent->>HiveMind: discover.js query
    HiveMind-->>NextAgent: Recent task outcomes
    NextAgent->>LightRAG: Graph query
    LightRAG-->>NextAgent: Entity relationships
    NextAgent->>NextAgent: Execute with enriched context

Usage

Basic Usage (Automatic)

No changes needed. Writeback happens automatically:

node ~/system/tools/mc.js done 8020 "Implemented blueprint liveness gate"

Console output:

Task #8020 marked as done
✓ Outcome recorded in HiveMind
✓ Queued for LightRAG ingest

With Evidence (Recommended)

node ~/system/tools/mc.js done 8020 \
  --evidence ~/system/evidence/my-validation/ \
  "All acceptance criteria met. Evidence in attached bundle."

Result: Evidence path included in intel metadata + outbox entry.

Bypass Proveo Gate (Emergency)

node ~/system/tools/mc.js done 8020 \
  --force "Production incident, validated live with CEO" \
  "Hotfix deployed"

Result: quality_gate: "bypassed" + force reason in metadata.

Verification

Check HiveMind Writeback

# List recent task outcomes
sqlite3 ~/system/databases/hivemind.db <<EOF
SELECT id, substr(content, 1, 80), created_at 
FROM intel 
WHERE source = 'mc-done' 
ORDER BY id DESC 
LIMIT 5;
EOF

Expected: Your recently completed task appears in top 5.

Check Outbox Queue

tail -n 5 ~/system/logs/mc-task-outcomes.jsonl

Expected: Last line is your task in JSON format.

Count pending:

wc -l < ~/system/logs/mc-task-outcomes.jsonl

Check LightRAG Ingest Status

curl -s http://localhost:9621/documents | jq '.statuses | {processed, pending, failed}'

Expected (after 03:00 cron):

End-to-End Test

# 1. Create test task
TEST_ID=$(node ~/system/tools/mc.js add "E2E writeback test" --owner john | grep -o '#[0-9]*' | tr -d '#')

# 2. Mark it done
node ~/system/tools/mc.js done $TEST_ID "Test outcome with unique marker $(date +%s)"

# 3. Check HiveMind (should appear within 1 second)
sqlite3 ~/system/databases/hivemind.db \
  "SELECT content FROM intel WHERE content LIKE '%E2E writeback test%' ORDER BY id DESC LIMIT 1;"

# 4. Check outbox (should appear immediately)
grep "$TEST_ID" ~/system/logs/mc-task-outcomes.jsonl

# 5. Check retrieval (next day after LightRAG ingest)
node ~/system/tools/discover.js "E2E writeback test"

Error Handling

HiveMind Write Failure

Symptom: Console shows:

[WARN] Failed to write task outcome to HiveMind: SQLITE_BUSY
Task #8020 still marked done, but intel not recorded.

Cause: Database locked (another agent writing).

Impact: Task marked done, but learning not immediately available.

Mitigation:

  1. Outbox still written → LightRAG ingest will capture it
  2. HiveMind write retried 3x with exponential backoff
  3. If all retries fail → logged to ~/system/logs/mc-errors.log

Recovery: Manual retry:

node ~/system/tools/mc.js writeback-retry 8020

Outbox Write Failure

Symptom:

[ERROR] Failed to append to mc-task-outcomes.jsonl: EACCES
Task marked done, HiveMind updated, but outbox NOT updated.

Cause: Permissions issue or disk full.

Impact: Task learning in HiveMind (immediate queries work) but NOT in LightRAG (graph queries miss it).

Recovery:

# Fix permissions
chmod 644 ~/system/logs/mc-task-outcomes.jsonl

# Backfill from HiveMind
node ~/system/tools/backfill-outbox.js --since "2026-04-16"

LightRAG Ingest Failure

Symptom (next day):

curl -s http://localhost:9621/documents | jq .statuses.failed
# Shows increase in failed count

Diagnosis:

docker logs lightrag | grep -i error | tail -20

Common causes:

Recovery:

  1. Check Neo4j health: docker logs neo4j --tail 50
  2. Restart LightRAG if needed: docker restart lightrag
  3. Re-submit failed docs: node ~/system/tools/lightrag-bulk-upload.js --retry-failed

Performance & Overhead

Latency Impact on mc.js done

Before writeback: mc.js done took ~50ms
After writeback: mc.js done takes ~120ms (+70ms)

Breakdown:

User perception: Negligible (< 200ms total).

Disk Usage

Outbox growth: ~500 bytes/task

Projection:

HiveMind growth: ~800 bytes/task

Projection:

LightRAG Ingest Load

Nightly bulk upload:

Current backlog: 63,359 pending (from research phase). Drain time: ~21 hours at 50 docs/min.

Strategy: Prioritize today's tasks (outbox JSONL) → backlog second.

Configuration

Disable Writeback (Not Recommended)

Environment variable:

export MC_DISABLE_WRITEBACK=true
node ~/system/tools/mc.js done 8020 "outcome"
# HiveMind + outbox writes skipped

Use case: Testing mc.js in isolation without side effects.

Change Outbox Path

Environment variable:

export MC_OUTBOX_PATH=/tmp/my-outbox.jsonl
node ~/system/tools/mc.js done 8020 "outcome"

Use case: Custom ingest pipelines.

Tune HiveMind Retry Logic

File: ~/system/tools/mc.js (around line 420)

const HIVEMIND_RETRIES = 3;           // Default 3
const HIVEMIND_RETRY_DELAY_MS = 500;  // Default 500ms

Increase if HiveMind frequently locked.

Integration with Other Systems

Session Logs Linkage

Writeback includes session_id (if available). Full conversation context in:

~/system/memory/sessions/2026-04-16-<session_id>.md

Use case: Debugging why agent made specific decision → trace back to full session.

Evidence Bundles

Writeback includes evidence_ref (if provided). Structure:

~/system/evidence/system-evolution-2026-04-16/
  ├── SUMMARY.md
  ├── v1-lightrag-health.json
  ├── v2-intel-tail.txt
  └── ...

Use case: Proveo validation requires evidence → MC done links to bundle → LightRAG indexes → future agents discover "how system-evolution was validated."

Blueprint Linkage

Writeback includes blueprint_ref (if task was created with --blueprint-ref).

Use case: Agent query "which tasks updated Plock blueprint?" → LightRAG returns graph of related tasks.

Troubleshooting

Issue 1: No Intel Appearing in HiveMind

Diagnosis:

# Check last HiveMind write
sqlite3 ~/system/databases/hivemind.db \
  "SELECT MAX(created_at) FROM intel WHERE source='mc-done';"

# Check mc.js error log
grep "HiveMind write" ~/system/logs/mc-errors.log | tail -10

Possible causes:

Issue 2: Outbox Growing But LightRAG Not Ingesting

Diagnosis:

# Check LightRAG ingest rate
curl -s http://localhost:9621/documents | jq '.statuses | {processed, pending}'

# Check cron job status
launchctl list | grep lightrag-bulk-upload

Possible causes:

Issue 3: Duplicate Intel Entries

Symptom:

sqlite3 ~/system/databases/hivemind.db \
  "SELECT COUNT(*) FROM intel WHERE content LIKE '%Task #8020%';"
# Returns: 3 (expected: 1)

Cause: Agent called mc.js done 8020 multiple times (idempotency bug).

Fix (immediate):

# Dedupe
sqlite3 ~/system/databases/hivemind.db <<EOF
DELETE FROM intel 
WHERE id NOT IN (
  SELECT MIN(id) FROM intel 
  WHERE source='mc-done' 
  GROUP BY json_extract(metadata, '$.task_id')
);
EOF

Fix (permanent): Add unique constraint (MC #8051):

CREATE UNIQUE INDEX idx_intel_mc_task ON intel (
  (json_extract(metadata, '$.task_id'))
) WHERE source = 'mc-done';

Best Practices

1. Always Provide Outcome Message

# BAD (generic)
node ~/system/tools/mc.js done 8020

# GOOD (specific)
node ~/system/tools/mc.js done 8020 \
  "Blueprint liveness gate implemented. mc.js now checks mtime. 10 existing blueprints audited."

Why: Outcome text flows to HiveMind → LightRAG → future agent queries. Generic "done" loses context.

node ~/system/tools/mc.js done 8020 \
  --evidence ~/system/evidence/my-validation/ \
  "All acceptance criteria met"

3. Tag Retroactively if Needed

Outbox uses auto-extracted tags, but you can add manual tags:

node ~/system/tools/mc.js update 8020 --tags "critical,production,hotfix"
node ~/system/tools/mc.js done 8020 "Emergency fix deployed"

Tags flow to HiveMind + outbox.

4. Review Outbox Weekly

# Check for stale entries (> 7 days, not ingested)
awk -F, '{print $1}' ~/system/logs/mc-task-outcomes.jsonl | \
  while read line; do
    TIMESTAMP=$(echo "$line" | jq -r .completed_at)
    # (date comparison logic)
  done

Action: If > 500 stale entries → investigate LightRAG pipeline.

Monitoring & Alerts

Key Metrics

Metric Command Threshold
Writeback success rate grep "✓ Outcome recorded" ~/system/logs/mc.log | wc -l > 95%
Outbox size wc -l ~/system/logs/mc-task-outcomes.jsonl < 10,000 lines
LightRAG ingest lag Compare outbox timestamp vs. LightRAG processed count < 48h

Alerting (future):

# Add to daily briefing
if [ $(wc -l < ~/system/logs/mc-task-outcomes.jsonl) -gt 5000 ]; then
  echo "WARN: MC outbox has 5000+ pending tasks. LightRAG ingest lagging."
fi

Future Enhancements

Planned (MC #8052)

  1. Smart tag extraction: Use LLM to extract domain tags ("fintech", "RAG", "mobile")
  2. Outcome quality scoring: Flag low-quality outcomes ("done" with no context)
  3. Session summary: Auto-generate 3-sentence summary from full session log

Under Consideration

Changelog

Date Version Change
2026-04-16 2.0.0 Auto-writeback implemented (MC #8020 Task 7)
2026-03-15 1.8.0 Added --evidence flag to mc.js done
2026-02-10 1.5.0 HiveMind integration

Questions? Contact Chip Huyen (AgentForge lead) or Petter Graff (team lead).

Blueprint Liveness Gate

Blueprint Liveness Gate

Owner: CodeCraft Implemented: 2026-04-16 (System Evolution Phase 2, MC #8026) Files: ~/system/tools/mc.js (migration + add handler + done gate)

Purpose

Blueprints drift. BUILD-BLUEPRINT.md files describe architecture, stack, integrations — but they're written once and then ignored. Six months later the code has migrated (Hono → Kotlin, Vite → Next.js) and the blueprint still shows the old stack. New agents reading the blueprint get misled.

The liveness gate forces a blueprint to be touched during the task that changes it. If a task is tagged with --blueprint-ref, it cannot be marked done unless the referenced blueprint file has been modified since the task started.

How it works

mc.js add "Switch Plock to Next.js" --blueprint-ref ALAI/products/plock/BUILD-BLUEPRINT.md
  → stores blueprint_ref in tasks table
mc.js start <id>                    → records started_at
(developer works, updates code + blueprint)
mc.js ready <id> "tested X"
mc.js done <id> "completed"
  → gate reads task.blueprint_ref, fs.statSync(bp).mtimeMs
  → if mtime < started_at → REJECT with ZAKON error
  → if mtime ≥ started_at → PASS

Usage

Add a task bound to a blueprint:

node ~/system/tools/mc.js add "Add GraphQL to Drop" \
  --blueprint-ref ALAI/products/drop/BUILD-BLUEPRINT.md \
  --priority H

Check which tasks have blueprint refs:

sqlite3 ~/system/databases/mission-control.db \
  "SELECT id, title, blueprint_ref FROM tasks WHERE blueprint_ref IS NOT NULL ORDER BY id DESC LIMIT 10"

Retrofit an existing task:

sqlite3 ~/system/databases/mission-control.db \
  "UPDATE tasks SET blueprint_ref='ALAI/products/plock/BUILD-BLUEPRINT.md' WHERE id=5126"

Emergency bypass

Use only when the blueprint genuinely shouldn't change (e.g. a typo fix, dependency bump that doesn't alter stated architecture).

node ~/system/tools/mc.js done <id> "outcome" --force "reason why blueprint unchanged"

The --force reason is logged to HiveMind for audit.

Adding a new blueprint

  1. Create <project>/BUILD-BLUEPRINT.md with sections: Scope, Stack, Ownership, Change Protocol, Last Updated, Related MC Tasks
  2. Path in --blueprint-ref is relative to $HOME (gate uses path.resolve(os.homedir(), task.blueprint_ref))
  3. Run bash ~/system/tools/zakon-plan-lint.sh <file> as a sanity check on the document structure (the linter accepts blueprints too)

Retrofit pattern (Plock example)

~/ALAI/products/plock/BUILD-BLUEPRINT.md got a ## 11. Stack Compliance section appended:

## 11. Stack Compliance

| Layer    | Current              | CEO Standard          | Status              |
|----------|----------------------|-----------------------|---------------------|
| Backend  | Kotlin + Ktor        | Kotlin + Ktor         | COMPLIANT           |
| Frontend | Vite MFE             | Next.js 15            | MIGRATION — MC #5126|
| DB       | PostgreSQL + Flyway  | PostgreSQL + Flyway   | COMPLIANT           |
| Testing  | JUnit + Playwright   | JUnit + Playwright    | COMPLIANT           |
| Monorepo | Turborepo + pnpm     | Turborepo + pnpm      | COMPLIANT           |

Do not refactor the whole blueprint at once — one compliance table is enough to surface the drift.

System Regression Suite (weekly)

System Regression Suite

Owner: Angie Jones / Proveo Implemented: 2026-04-16 (System Evolution Phase 3, MC #8036) Script: ~/system/tools/system-regression.sh

Purpose

Catch system drift early. The ALAI "system that builds systems" accumulates state — LightRAG ingest, HiveMind intel, daemons, blueprints — and every small change risks breaking a seam. The regression suite runs 10 checks in under 10 seconds and exits non-zero if any critical component regresses.

The 10 Checks

# Check Tool PASS condition
1 Tools health discover.js --verify exit 0
2 MC smoke mc.js list --limit 1 exit 0
3 LightRAG docker docker inspect lightrag .State.Health.Status == "healthy"
4 LightRAG HTTP curl /health JSON with .status field
5 HiveMind readable sqlite3 … SELECT 1 exit 0
6 HiveMind growing count vs baseline count ≥ baseline
7 Outbox exists test -f mc-task-outcomes.jsonl file present
8 ZAKON PLAN drift zakon-plan-lint.sh *-plan.md WARN if any FAIL, not total fail
9 Dead daemons launchctl list count of exit≠0 in ALAI namespace < 5
10 Cost tracker cost-tracker.js summary today Input tokens ≥ 0 (non-error)

Runtime target: < 2 minutes. Measured: 9 seconds on an idle machine.

Reading the output

Summary line at end:

FAILED=X, WARNINGS=Y, PASS=Z, TOTAL=10

Exit codes:

Usage

Manual run

bash ~/system/tools/system-regression.sh
# exit code visible in $?

Scheduled (weekly via launchd)

Create ~/Library/LaunchAgents/com.alai.system-regression.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key><string>com.alai.system-regression</string>
    <key>ProgramArguments</key>
    <array>
        <string>/bin/bash</string>
        <string>/Users/makinja/system/tools/system-regression.sh</string>
    </array>
    <key>StartCalendarInterval</key>
    <dict>
        <key>Weekday</key><integer>1</integer>
        <key>Hour</key><integer>6</integer>
        <key>Minute</key><integer>0</integer>
    </dict>
    <key>StandardOutPath</key><string>/Users/makinja/system/logs/system-regression.log</string>
    <key>StandardErrorPath</key><string>/Users/makinja/system/logs/system-regression.err</string>
</dict>
</plist>

Load with:

launchctl load ~/Library/LaunchAgents/com.alai.system-regression.plist
launchctl start com.alai.system-regression

Schedule: every Monday 06:00 CEST. Output tailed to ~/system/logs/system-regression.log.

Baseline management

Checks 6 (HiveMind growth) stores /tmp/regression-baseline-hivemind.txt. If /tmp gets cleared (reboot), the next run re-bootstraps. This is intentional — growth check is soft-enforced, warns on first run, tracks from second onwards.

If you want a durable baseline, move the file:

mkdir -p ~/system/state
mv /tmp/regression-baseline-hivemind.txt ~/system/state/
# then edit the script: BASELINE_FILE="$HOME/system/state/regression-baseline-hivemind.txt"

Adding a new check

  1. Open ~/system/tools/system-regression.sh
  2. Find the # --- Check N --- pattern
  3. Copy a block, increment counter, write the check
  4. Update TOTAL at bottom
  5. Re-run; confirm new check shows up in output
  6. Commit + update this runbook

Keep checks fast (< 2s each) and independent (no check should depend on another passing).

Known current failures (2026-04-16)

These fail but are tracked as pre-existing debt, not regressions:

System Evolution 2026-04-16 — Main Runbook

ALAI System Evolution — April 2026 Upgrade

Date: 2026-04-16
Team Lead: Petter Graff
Contributors: Chip Huyen, Martin Kleppmann, Angie Jones, Kelsey Hightower
Status: Complete — Evolution Score 3/10 → 7/10
Mission Control: Task #8020 (master)

Executive Summary

The ALAI system was designed to be self-improving, but critical feedback loops were broken. This upgrade repairs three core chains:

  1. Knowledge chain: Task completion now flows → HiveMind → LightRAG → agent retrieval (default-on)
  2. State chain: Ghost databases removed, single source of truth enforced
  3. Governance chain: ZAKON PLAN linter + Proveo gate + Blueprint liveness enforced at commit/done time

Before: System ingested knowledge but never retrieved it. Agents hallucinated. Plans were shipped without validation tasks.
After: Every task enriches the next one. Every plan is enforced. Every "done" requires evidence.

Architecture: Self-Improving Loop

flowchart LR
    A[Task Done] -->|mc.js done| B[HiveMind Write]
    B -->|mc-task-outcomes.jsonl| C[Outbox Queue]
    C -->|lightrag-bulk-upload.js| D[LightRAG Ingest]
    D --> E[Neo4j Graph + Entity Index]
    E -->|discover.js default| F[Agent Query]
    F -->|Enhanced Context| G[Next Task Smarter]
    G --> A
    
    style A fill:#e1f5e1
    style D fill:#fff3cd
    style F fill:#d1ecf1
    style G fill:#e1f5e1

Key insight: The loop was 25% complete (ingest only). Now it's 90% (ingest → index → retrieve → apply → writeback).

What Changed — 11 Core Improvements

1. LightRAG Health Probe Fix

File: ~/system/docker/lightrag/docker-compose.yml (line 74)

Problem: Probe used curl, but image had no curl binary. Container marked unhealthy for 46+ hours while pipeline worked.

Fix: Python probe using urllib.request:

healthcheck:
  test: ["CMD-SHELL", "python3 -c 'import urllib.request; urllib.request.urlopen(\"http://localhost:9621/health\", timeout=5)' || exit 1"]
  interval: 15s
  timeout: 10s
  retries: 3
  start_period: 30s

Verification:

docker inspect lightrag | jq -r '.State.Health.Status'
# Expected: "healthy"

Known issue: Under heavy ingest load (87,000+ docs), probe can timeout. Container remains functional. Recommend 30s timeout for production.

2. Ghost HiveMind Symlink

Files:

Problem: Subagents referencing old path wrote to void. Silent intel loss.

ln -sf ~/system/databases/hivemind.db ~/.claude/hivemind.db

Verification:

ls -lah ~/.claude/hivemind.db
# Expected: lrwxr-xr-x ... -> /Users/makinja/system/databases/hivemind.db

sqlite3 ~/.claude/hivemind.db "SELECT COUNT(*) FROM intel;"
# Expected: 30912 (matches real DB)

3. LightRAG Default-On in discover.js

File: ~/system/tools/discover.js

Problem: LightRAG was flag-gated (if (flags.lightrag)). Default agent workflow never queried the graph. 68,602 documents ingested but zero retrieval.

Fix: Inverted logic:

// OLD: const useLightRAG = flags.lightrag;
// NEW:
const useLightRAG = !flags['no-lightrag'];

LightRAG now runs by default with 5s timeout fallback. Opt-out: discover.js --no-lightrag "query".

Verification:

node ~/system/tools/discover.js "MC task workflow" | grep -q "LightRAG"
# Expected: LightRAG section in output

Runbook: See ~/system/docs/runbooks/lightrag-default-on.md

4. Auto-Writeback on mc.js done

Files:

Problem: Task learnings stayed in session logs. Never indexed. Next agent started from zero context.

Fix: When mc.js done <id> runs:

  1. Extracts task summary + outcome
  2. Writes to HiveMind (intel table) — fire-and-forget
  3. Appends to JSONL outbox for bulk LightRAG ingest
  4. Non-blocking: HiveMind failure logs error but doesn't block task closure

Format (outbox):

{
  "task_id": 8020,
  "title": "System Evo T11: Blueprint liveness gate",
  "outcome": "Gate implemented. mc.js checks blueprint mtime during done.",
  "timestamp": "2026-04-16T21:12:03Z",
  "tags": ["mc", "blueprint", "governance"]
}

Verification:

tail -n 3 ~/system/logs/mc-task-outcomes.jsonl
sqlite3 ~/system/databases/hivemind.db \
  "SELECT COUNT(*) FROM intel WHERE category='briefing' AND created_at > datetime('now', '-1 hour');"

Runbook: See ~/system/docs/runbooks/mc-done-auto-writeback.md

5. ZAKON PLAN Linter

File: ~/system/tools/zakon-plan-lint.sh

Problem: Plans often shipped without validation task (Proveo/Angie) or documentation task (Skillforge). Hard Constraint violation was voluntary.

Fix: Pre-commit hook enforces ZAKON PLAN:

bash ~/system/tools/zakon-plan-lint.sh ~/system/specs/my-plan.md
# Exit 0: Plan compliant (has validation + docs task)
# Exit 1: Plan missing required tasks

Detects:

Verification:

# Test with compliant plan
bash ~/system/tools/zakon-plan-lint.sh ~/system/specs/system-evolution-plan.md && echo PASS

# Regression suite runs linter on all specs/*-plan.md (max 10)
bash ~/system/tools/system-regression.sh | grep "ZAKON PLAN"

Runbook: See ~/system/docs/runbooks/zakon-plan-linter.md

6. Proveo Gate in mc.js done

File: ~/system/tools/mc.js (done command)

Problem: Builder could mark task done without validation evidence. Hard Constraint #4 ("Builder cannot say done") was unenforced.

Fix: mc.js done <id> now checks:

  1. Does task have evidence_ref field populated?
  2. Was last update by Proveo/Angie agent?
  3. If neither → reject unless --force "reason"

Force reason logged to HiveMind with quality gate flag.

Usage:

# Normal flow (requires evidence)
node ~/system/tools/mc.js done 8020

# Emergency bypass (logged + flagged)
node ~/system/tools/mc.js done 8020 --force "Production incident, validated live with CEO"

Verification: Evidence file: ~/system/evidence/system-evolution-2026-04-16/v4-reject.txt + v4-accept.txt

7. Blueprint Liveness Gate

Files:

Problem: Blueprints were static documentation. Tasks claiming "stack migration complete" never updated blueprint. Plock blueprint was 40 days stale despite Vite→Next.js migration.

Fix:

  1. MC tasks can reference blueprint: mc.js add --blueprint-ref ~/ALAI/products/plock/BUILD-BLUEPRINT.md
  2. On mc.js done, gate checks: was blueprint file modified during task execution?
  3. If not → warn or reject (based on config)

Shared-configs blueprint: Created ~/felles/shared-configs/BUILD-BLUEPRINT.md covering:

Verification:

# Check Plock blueprint freshness
ls -l ~/ALAI/products/plock/BUILD-BLUEPRINT.md

# Verify shared-configs blueprint exists
cat ~/felles/shared-configs/BUILD-BLUEPRINT.md | head -20

Runbook: See ~/system/docs/runbooks/blueprint-liveness.md

8. Cost Tracker Token Counting Fix

Files:

Problem: Cost tracker reported 0 tokens despite active LLM usage. Alem had no spend visibility.

Root cause: Ollama adapter wasn't parsing response format correctly.

Fix: Updated adapter to handle Ollama /api/chat response structure + fallback for missing usage field.

Verification:

node ~/system/tools/cost-tracker.js summary today
# Expected: tokens_total > 0

# Evidence: ~/system/evidence/system-evolution-2026-04-16/v6-cost.txt
# Shows: Total requests: 10, token sample: 1,463

9. Regression Suite

File: ~/system/tools/system-regression.sh

Why: No automated smoke tests for system toolset. Breakage discovered days later by agents failing mid-task.

Coverage (10 checks, <10s runtime):

  1. Tools health (discover.js --verify)
  2. MC smoke (mc.js list --limit 1)
  3. LightRAG container health
  4. LightRAG HTTP reachable
  5. HiveMind readable
  6. HiveMind growing (delta check vs baseline)
  7. MC outbox exists
  8. ZAKON PLAN compliance scan (specs/*-plan.md)
  9. Dead daemon count (< 5 threshold)
  10. Cost tracker non-zero tokens

Output format: PASS / FAIL / WARN with color-coded summary.

Verification:

bash ~/system/tools/system-regression.sh
# Evidence: ~/system/evidence/system-evolution-2026-04-16/v8-regression.txt

Runbook: See ~/system/docs/runbooks/system-regression-suite.md

10. Orchestration Surface Authority

File: ~/system/rules/orchestration-surface.md

Problem: Three competing orchestration surfaces (Ollama DAG, Claude chains, PI factory) with no routing authority. Agents chose arbitrarily.

Fix: Decision table created:

Task type Surface Primary tool
Long-running DAG (> 5 min) Ollama DAG orchestrator-http-server.js
Interactive subagent (in-session) Claude chains YAML from ~/system/agents/chains/
Persistent company agent PI factory agent-factory.js
One-shot atomic build (< 10 min) Task tool (Agent) subagent_type param
Cron / scheduled CronCreate skill cron registry

Default when unsure: One-shot Task tool.

Verification: Evidence file: ~/system/evidence/system-evolution-2026-04-16/v7-orch.txt

11. Database Deduplication

Problem: Three MC database files found:

Agents using wrong path → empty queue, silent failures.

Fix: Empty duplicates removed. Only mission-control.db remains.

Verification:

ls -lh ~/system/databases/mission-control.db
ls -lh ~/system/tools/mc.db 2>/dev/null || echo "Correctly deleted"

Evidence: ~/system/evidence/system-evolution-2026-04-16/v7-dupes-gone.txt

Known Issues & Limitations

1. LightRAG Probe Timeout Under Load

Status: Non-critical
Symptoms: Health check times out during bulk ingest (87K+ docs)
Workaround: Container remains functional. Probe timeout doesn't affect pipeline.
Fix plan: Increase probe timeout to 30s in production (MC #8048)

2. B2 Offsite Backup Daemon Dead

Status: CRITICAL
Task: MC #5 (restart + fix)
Impact: No offsite backups since 2026-04-14

3. 43 Dead Daemons

Status: Fleet degraded
Task: MC #8049 (triage + restart priority)
List: launchctl list | awk '$1 == "-" && $2 != "0"'

4. ZAKON PLAN Compliance: 2/10 Historic Plans

Status: Expected drift
Action: Linter enforces NEW plans. Retro-fix not required.

Validation Evidence

All evidence stored in: ~/system/evidence/system-evolution-2026-04-16/

Check File Result
LightRAG health v1-lightrag-health.json Functional (probe timeout during load)
Auto-writeback v2-intel-tail.txt + v2-outbox-tail.txt 6 new intel entries
ZAKON linter v3-pass.txt + v3-fail.txt Detects missing tasks
Proveo gate v4-accept.txt + v4-reject.txt Rejects without evidence
Regression suite v8-regression.txt 7/10 PASS, 3 FAIL (expected)
Cost tracker v6-cost.txt Non-zero tokens (1,463 sample)
DB dedup v7-dupes-gone.txt Duplicates removed
Orchestration v7-orch.txt Authority doc created
Symlink v7-symlink.txt Ghost DB now symlinked

How to Verify System Health Post-Upgrade

Quick Check (30 seconds)

bash ~/system/tools/system-regression.sh
# Expected: 7+ checks PASS

Detailed Validation

1. LightRAG pipeline working:

curl -s http://localhost:9621/documents | jq '.statuses | {processed, pending, failed}'
docker inspect lightrag | jq -r '.State.Health.Status'

2. HiveMind auto-writeback:

# Create test task
TEST_ID=$(node ~/system/tools/mc.js add "Test writeback" --owner john | grep -o '#[0-9]*' | tr -d '#')

# Complete it
node ~/system/tools/mc.js done $TEST_ID

# Check intel table
sqlite3 ~/system/databases/hivemind.db \
  "SELECT content FROM intel WHERE content LIKE '%Test writeback%' ORDER BY id DESC LIMIT 1;"

3. ZAKON PLAN linter:

# Test with non-compliant plan (should fail)
echo "# Plan\nSome tasks but no validation" > /tmp/bad-plan.md
bash ~/system/tools/zakon-plan-lint.sh /tmp/bad-plan.md && echo "ERROR: should have failed"

# Test with system-evolution-plan (should pass)
bash ~/system/tools/zakon-plan-lint.sh ~/system/specs/system-evolution-plan.md && echo PASS

4. Proveo gate:

# Try to mark task done without evidence (should reject)
node ~/system/tools/mc.js done <task-without-evidence>
# Expected: Error message about missing validation

5. Cost tracker:

node ~/system/tools/cost-tracker.js summary today | jq .tokens_total
# Expected: > 0

Impact Metrics

Metric Before After Target
Evolution score 3/10 7/10 7/10 ✅
LightRAG ingest rate 5% 95%+ >95% ✅
LightRAG default retrieval 0% 100% 100% ✅
Dead daemons 12 43 <3 ⚠️
ZAKON PLAN compliance Partial 100% (new) 100% ✅
Self-test coverage ~15% 40%+ 40% ✅
Ghost databases 3 0 0 ✅
Auto-writeback No Yes Yes ✅

Overall: 7/9 targets met. Dead daemons (43) and B2 backup require follow-up (MC #8049, #5).

Next Steps

Immediate (CEO approved)

  1. Restart B2 backup daemon (MC #5) — CRITICAL
  2. Triage 43 dead daemons (MC #8049) — HIGH priority
  3. Monitor LightRAG ingest rate — Daily check for 1 week

Short-term (2 weeks)

  1. Retrofit Plock blueprint with stack compliance checklist
  2. LightRAG probe timeout increase to 30s in docker-compose.yml
  3. Weekly regression suite scheduled via launchd

Long-term (1 month)

  1. Extend ZAKON linter to check for Evidence Level (L2+ minimum)
  2. Blueprint liveness — change from warn to block
  3. HiveMind outbox idempotency — add unique constraint on correlation_id

Validated by: Angie Jones (Proveo) — Task #8027
Documented by: Skillforge — Task #8038
Approved by: Petter Graff (Team Lead)
Date: 2026-04-16 23:14 CEST

"Every task completion now enriches the next one. That is the evolution the CEO asked for." — Petter Graff

Hive Activation 2026-04-17 — Main Runbook

Hive Activation — 2026-04-17

Status: Phase 1–5 builders complete; Phase 6 validation in progress. Plan: ~/system/specs/hive-activation-plan.md Evidence: ~/system/evidence/hive-activation-2026-04-17/ Prior sprint: System Evolution 2026-04-16 (see system-evolution-2026-04-16.md).

Why this sprint

After System Evolution we knew:

Hive Activation is the follow-up: turn inventory into interaction.

End-state diagram

sequenceDiagram
  participant Agent as Any agent
  participant MC as mission-control.db
  participant HM as hivemind.db
  participant Subs as subscriptions
  participant Auto as hive-handlers/*.sh
  participant NewMC as auto-created MC task
  participant LO as learning-opportunities/

  Agent->>MC: mc.js done <id>
  MC->>HM: post learning (T7)
  MC->>HM: post task-completion (T2)
  MC->>HM: post failed-task (T12, if outcome/reason matches failure regex)

  HM-->>Subs: SELECT WHERE kind=... AND enabled=1
  Subs-->>Auto: spawn callback (fire-and-forget, non-blocking)

  Auto->>NewMC: mc.js add with dedup (proveo QA / skillforge BookStack / codecraft bug)
  Auto->>LO: write lesson draft (for failed-task)

  Note over NewMC: original mc.js done returned long ago
  Note over LO: human reviews drafts

What changed

Phase 1 — Event bus live (the spine)

Phase 2 — Library activation

Phase 3 — Skill usage visibility

Phase 4 — Discover

Phase 5 — Meta-agent activation

Active subscriptions (after Phase 1)

Agent Kind Handler
proveo task-completion hive-handlers/proveo-auto-qa.sh → QA review: #<id>
skillforge architecture-change hive-handlers/skillforge-auto-doc.sh → Update BookStack: <bp>
codecraft error hive-handlers/codecraft-auto-bug.sh → Investigate error intel#<id>
learning-agent learning (filter: FAILED) learning-opportunity-draft.sh → markdown draft

New daemons (launchd)

Label Schedule Script
com.alai.library-sync every 5 min library-sync-wrapper.sh (library sync + discover rebuild)
com.alai.skill-audit Monday 07:00 skill-audit-report.sh
com.alai.meta-agent-loop daily 03:30 meta-agent-loop.js

New knobs + files

Known issues (deliberate, not silent)

HiveMind Subscriptions

HiveMind Subscriptions

Owner: CodeCraft Implemented: 2026-04-17 (Hive Activation Phase 1) Code: ~/system/agents/hivemind/hivemind.js DB: ~/system/databases/hivemind.db (subscriptions table)

Purpose

Before Hive Activation, agents posted knowledge to HiveMind but nothing reacted. Subscriptions turn HiveMind into an event bus: an agent declares "when intel with kind=X arrives, run this callback", and on every post the callback fires async.

Data model

subscriptions(
  agent TEXT NOT NULL,
  kind TEXT NOT NULL,               -- matches intel.type
  callback TEXT NOT NULL,           -- shell command; event JSON piped to stdin
  enabled INTEGER DEFAULT 1,
  correlation_filter TEXT,          -- optional substring filter on message
  created_at TEXT,
  PRIMARY KEY (agent, kind, callback)
)

CLI

# Register
node ~/system/agents/hivemind/hivemind.js subscribe <agent> \
  --kind <type> \
  --callback "<shell cmd reading JSON from stdin>"

# List
node ~/system/agents/hivemind/hivemind.js subscriptions
node ~/system/agents/hivemind/hivemind.js subscriptions --agent proveo

# Disable (keeps row, sets enabled=0)
node ~/system/agents/hivemind/hivemind.js unsubscribe <agent> --kind <type>

# Dev: re-fire a stored intel row against all matching subscribers (useful for smoke tests)
node ~/system/agents/hivemind/hivemind.js fire <intel_id>

Callback semantics

Current seed subscriptions

Agent Kind Callback outcome
proveo task-completion Creates MC task QA review: #<src>
skillforge architecture-change Creates MC task Update BookStack: <blueprint>
codecraft error Creates MC task Investigate error intel#<id> (priority H)
learning-agent learning (filter: FAILED) Writes lesson draft to ~/system/learning-opportunities/

Handler scripts live in ~/system/tools/hive-handlers/ and include dedup logic so repeated events don't flood MC.

Adding a new subscription

  1. Write a handler script that reads JSON from stdin, does ONE small thing, exits 0. Keep it < 50 lines.
  2. Always include dedup — check MC or a file marker before creating a new task.
  3. Never block — worst case your callback failing should be a log line, not a dropped event for everyone else.
  4. Register: subscribe <your-agent> --kind <type> --callback "bash <path-to-handler>"
  5. Smoke: post test-agent <type> "smoke payload" and verify handler fired.
  6. Document in this file.

Emergency disable

Stop all subscriber firing without touching the table:

sqlite3 ~/system/databases/hivemind.db "UPDATE subscriptions SET enabled=0 WHERE agent != 'learning-agent';"

Re-enable selectively with enabled=1 by PK.

Failure modes seen

Library Auto-Push

Library Auto-Push

Owner: FlowForge Implemented: 2026-04-17 (Hive Activation Phase 2 T4) Plist: ~/Library/LaunchAgents/com.alai.library-sync.plist Wrapper: ~/system/tools/library-sync-wrapper.sh

Purpose

Before this daemon, library.js sync was manual. Result: 26-day-old distributions. Every company ran skills that diverged from the global source.

Now library.js sync --fix runs every 5 min, and immediately after, discover.js --rebuild-index refreshes the search index.

Schedule

Wrapper flow

node ~/system/tools/library.js sync --fix   # distribute skills across companies
node ~/system/tools/discover.js --rebuild-index   # refresh search index

Pre-first-run snapshot

Before the daemon's very first run, a snapshot of every company's .claude/skills/ was saved to:

~/system/backups/library-pre-autopush-20260417-0041/

If auto-push ever overwrites a legitimate company customization, restore from there.

Operation

# Status
launchctl list | grep library-sync
# LastExitStatus should be 0

# Force a run
launchctl start com.alai.library-sync

# Tail log
tail -50 ~/system/logs/library-sync.log

# Current sync state
node ~/system/tools/library.js status

Drift handling

library.js sync --fix will flag company overrides (the company's version of a skill differs from global). Current drift at first-run: 8 items across CodeCraft + Lexicon. This is expected — these are intentional overrides, not bugs. The daemon logs them but doesn't force-overwrite.

If you WANT to force-normalize a company: node ~/system/tools/library.js push <skill> --company <name>.

Rollback procedure

If auto-push corrupts company skills:

launchctl unload ~/Library/LaunchAgents/com.alai.library-sync.plist
# restore from snapshot
cp -a ~/system/backups/library-pre-autopush-20260417-0041/<company>/ ~/ALAI/<company>/.claude/skills/
# or selective restore per skill

Re-enable: launchctl load the plist.

Known issues

Skill Use Counter

Skill Use Counter

Owner: CodeCraft Implemented: 2026-04-17 (Hive Activation Phase 3 T7 + T8) Hook: ~/.claude/hooks/skill-use-counter.sh Viewer: ~/system/tools/skill-usage.js Audit: ~/system/tools/skill-audit-report.sh (weekly)

Purpose

Before T7, skill-registry.db.use_count stayed at 0 forever. We had 76 skills on disk and no idea which were alive, which were dead, or which to retire.

Now: every Skill tool invocation increments use_count. Once a week an audit report lists top used + dead candidates.

Hook wiring

Registered in ~/.claude/settings.json under PostToolUse:

{
  "matcher": "Skill|mcp__skill",
  "hooks": [
    {"type": "command", "command": "bash ~/.claude/hooks/skill-use-counter.sh", "async": true}
  ]
}

The hook reads the tool invocation JSON from stdin, extracts the skill name, runs:

UPDATE skills SET use_count = use_count + 1 WHERE name = '<skill>';

Never blocks: exits 0 even if the DB is missing. SQL injection safe (single quotes are doubled before the sqlite3 call).

Logs

Every increment appends a line to ~/system/logs/skill-use.log:

2026-04-17T00:42:13Z SKILL=sync
2026-04-17T00:43:01Z SKILL=build

Reading usage

# Top 20 used
node ~/system/tools/skill-usage.js

# All (including 0-count)
node ~/system/tools/skill-usage.js --all

# Retirement candidates (0 uses, > 30 days old)
node ~/system/tools/skill-usage.js --dead

# Custom window
node ~/system/tools/skill-usage.js --dead --days 60

Weekly audit

com.alai.skill-audit.plist runs every Monday 07:00 CEST, writes ~/system/reports/skill-audit-<date>.md with:

Week 1 is bootstrap only — every skill was registered at use_count=0, so all show up as candidates. Wait for week 2+ to make real retirement decisions.

What to do with retirement candidates

Three options — pick per skill:

  1. Keep — valuable but under-discovered. Improve its description/trigger in the registry so agents find it.
  2. Deprecate — keep file on disk for reference but hide from discovery: UPDATE skills SET active=0 WHERE name='X'.
  3. Remove — delete directory + registry row. Only if truly obsolete.

The audit report is read-only. No auto-retirement.

Known issues

Meta-Agent Loop

Meta-Agent Loop

Owner: AgentForge Implemented: 2026-04-17 (Hive Activation Phase 5 T11) Script: ~/system/tools/meta-agent-loop.js Schedule: ~/Library/LaunchAgents/com.alai.meta-agent-loop.plist — daily 03:30 CEST

Purpose

Before T11, nothing converted recurring lessons into new skills. If CodeCraft fixed the same class of bug three times, the third fix wasn't easier than the first.

The meta-agent loop reads HiveMind learning + failed-task intel from the last 24 hours, detects themes appearing 3+ times, and files a NEW SKILL PROPOSAL MC task for Alem to review.

Algorithm (intentionally simple)

  1. Query intel in last 24h where type IN ('learning', 'failed-task').
  2. Tokenize each message, lowercase, drop stopwords + tokens shorter than 4 chars.
  3. Build bigram frequency map — for each pair of consecutive tokens, count the distinct intel rows it appears in.
  4. Take top 5 bigrams where distinct-row count ≥ 3.
  5. For each theme: check MC for an existing NEW SKILL PROPOSAL: <theme> (dedup across days). If absent, create.

Intentionally NOT doing: embedding clustering, LLM classification. Keep it legible; upgrade if bigram noise becomes real.

MC proposal shape

Approval gate — hard rule

The loop never commits or pushes skills. Only proposes in MC. Alem reviews, approves, then a human (or skill-creator agent) runs node ~/system/tools/library.js push <skill>. This is deliberate: meta-learning without human-in-the-loop is how systems drift.

Manual run

node ~/system/tools/meta-agent-loop.js 2>&1 | tail -20

Idempotent. Re-running the same day won't create duplicates (MC dedup check).

Tuning knobs (in the script)

Known issues

discover.js Re-Index + LightRAG Fallback

discover.js — Re-Index + LightRAG Fallback

Owner: AgentForge Implemented: 2026-04-17 (Hive Activation Phase 4 T9 + T10) Tool: ~/system/tools/discover.js Index: ~/system/tools/.alai/discover-index.json Post-sync wrapper: ~/system/tools/library-sync-wrapper.sh

Purpose

Before T9, discover.js "drop" returned 3 product hits and 0 hits across tools/skills/agents/MCP/BookStack. Index was slow and shallow.

After T9+T10: persistent inverted index (521 entries from 6 sources), sub-50ms queries, and a semantic LightRAG fallback when the local index is thin.

Sources indexed

Source Count (first build) Origin file
tools 206 ~/system/tools/manifest-index.md + manifest.md
skills 64 ~/system/databases/skill-registry.db
agents 22 ~/system/agents/specialist-mapping.json + ~/.claude/agents/*.md
mcp 7 ~/.claude.json .mcpServers
bookstack 182 ~/system/config/bookstack-sync-map.json
products 40 ~/system/data/product-index.json

Rebuild

# Manual
node ~/system/tools/discover.js --rebuild-index

# Automatic — happens every 5 min via library-sync-wrapper.sh,
# which is what com.alai.library-sync plist invokes after library sync

Atomic writes: index is written to .tmp, then renamed. No partial state visible to readers.

Query behavior

node ~/system/tools/discover.js "<query>"
# → tokens match inverted index → grouped by source

node ~/system/tools/discover.js "<query>" --no-lightrag
# → suppress LightRAG fallback entirely

If total hits across (tools + skills + agents + mcp + bookstack) < 3, the script queries LightRAG with a 5-second timeout. Results are prefixed LIGHTRAG (fallback — semantic) so you can tell them apart from keyword matches.

If LightRAG is slow or unavailable, the fallback silently times out and returns whatever the local index had. No hang.

Known issues

Tender Parking Protocol

Tender Parking Protocol

Owner: John (ALAI Holding) Updated: 2026-04-17 Decision: Alem, 2026-04-17 11:30 CEST

Policy

Tenders auto-ingested from TED/Mercell are NOT auto-worked. They are parked by default and selectively reactivated when:

  1. Strategic fit — deadline ≥ 14 days, budget signal matches ALAI capacity, domain alignment (fintech / AI services / ICT consulting)
  2. Capacity window — team has open slot AND no higher-priority client work
  3. Explicit CEO trigger — Alem escalates a specific notice to "live" state

Tenders are never auto-closed — one of them will activate eventually.

State model

[TENDER] ingested (open, H priority)
    │
    ├─── Alem picks → resume → assign to Proxima/Lexicon → lead
    │
    └─── Capacity window expires → paused (parked)

Paused tenders retain all their metadata (TED link, Mercell URL, deadline, score). No information lost.

Bulk operations

# Park ALL open tenders in one pass
for id in $(node ~/system/tools/mc.js list --status open 2>&1 | grep -iE "\[TENDER\]" | grep -oE "^#[0-9]+" | tr -d '#'); do
  node ~/system/tools/mc.js pause $id --reason "Parkirano — tender-parking-protocol.md" --actor alem
done

# Count paused tenders
node ~/system/tools/mc.js list --status paused 2>&1 | grep -cE "\[TENDER\]"

# Reactivate a specific tender
node ~/system/tools/mc.js resume <id> --actor alem
node ~/system/tools/mc.js priority <id> H

Reactivation criteria (selection checklist)

When deciding which parked tender to resume:

If 4+ ✓ → resume. If < 4 ✓ → leave parked or close permanently.

Historical snapshot — 2026-04-17 mass parking

54 [TENDER] tasks parked in one operation at 11:30 CEST. All entries carry notice IDs and TED/Mercell links in their description. Sample:

Ownership

Change log

Email Address Validation (Pre-Send Gate)

Email Address Validation (pre-send gate)

Owner: CodeCraft (tooling) Implemented: 2026-04-18 (after Quran outreach 2-bounce incident) Script: ~/system/tools/email-address-validate.js Cache: ~/system/databases/email-address-cache.sqlite (7-day TTL)

Why

First-send to uncatalogued institutional addresses (Al-Burhan, FIN Sarajevo) bounced because of typo/wrong-user assumptions. Adding an MX-lookup gate in front of SMTP send catches that class of failure before a message leaves the building.

What it does

Layer Catches Cost
Syntax check (RFC 5322 simplified) invalid@ or missing domain negligible
DNS MX lookup Nonexistent domain, missing MX records ~50–200 ms first time, cached after
SMTP RCPT probe (optional, --probe) Hard 550 rejections on strict servers ~1–5 s
Cache Repeat validation on known addresses 0 ms

What it does NOT catch

Gmail-hosted domains respond 250 accepted to RCPT even for nonexistent recipients. The real NDR arrives seconds/minutes later from the submission pipeline. Since many academic/institutional domains are on Google Workspace, RCPT probing is not a reliable filter for the class of errors we hit.

Mitigation: the validator prints a WARN when sending to a first-seen Gmail-hosted address, instructing the caller to verify manually.

Integration

mail-native.js sendEmail() calls the validator before the Himalaya/SMTP path. Hard-block on exit=1 (no MX / syntax). --force bypasses.

CLI

node ~/system/tools/email-address-validate.js <email>          # syntax + MX, cached
node ~/system/tools/email-address-validate.js <email> --probe  # + SMTP RCPT (see caveat above)
node ~/system/tools/email-address-validate.js <email> --force  # skip cache

Exit codes: 0 valid, 1 invalid, 2 transient/unknown.

LightRAG Backup (Azure-native + local safety net)

LightRAG Backup (Azure-native + local safety net)

Domain note (2026-05-17): References to lightrag.basicconsulting.no in this doc are the legacy hostname. Current live endpoint: lightrag.alai.no.

Operational update — 2026-08-19 (MC #107355)

This section supersedes the older Azure-primary statements retained below for migration history. The live source is now the local compose project at ~/system/lightrag-local/docker-compose.local.yml, backed by the seven lr106747-* volumes. The Cloudflare dashboard route for lightrag.alai.no targets this machine's localhost:9621; com.john.cloudflared owns the tunnel process.

The 2026-08-16 weekly backup stopped LightRAG and Neo4j, but LightRAG start raced Neo4j health. docker compose start lightrag failed, the script reduced that to a warning, continued through offsite upload, and reported Done, leaving the public endpoint at HTTP 502 until 2026-08-19.

The restart contract is now fail-closed:

  1. retry start of the existing Neo4j container;
  2. wait up to 180 seconds for lightrag-neo4j health;
  3. retry start of the existing LightRAG container;
  4. wait up to 300 seconds for lightrag health;
  5. exit non-zero with operator-action text if either service fails recovery.

This path uses docker compose start; it does not recreate containers, delete volumes, or restore data. Public recovery was verified through browser HTTP 200 plus wrapper status, hybrid query, and graph explore. Evidence: /Users/makinja/system/evidence/107355/REPORT.md.

Remaining validation: run the next scheduled full backup under observation to prove the new gates end to end. Do not trigger an extra 1.5 GB backup solely as a test because it intentionally causes production downtime.

Owner: FlowForge (infra) Implemented: 2026-04-18 (updated for Azure migration 2026-04-18) Source of Truth: Azure VM vm-alai-lightrag (20.240.61.67) Schedule: Weekly Sunday 04:00 CEST Script: ~/system/tools/lightrag-backup.sh (SSH-based) Plist: ~/Library/LaunchAgents/com.alai.lightrag-backup.plist Azure creds: ~/system/config/azure-lightrag-backup.env (mode 0600)

What is backed up

4 Docker volumes (+ checksums + README):

Volume Content Typical size
lightrag-data LightRAG KV store + inputs ~300 MB
lightrag-kg Knowledge graph files small
lightrag-cache LLM response cache small
lightrag-neo4j-data Neo4j graph entities + relations ~170 MB

Typical total: 500 MB – 1 GB compressed.

How it runs (POST-MIGRATION)

Source: Azure VM vm-alai-lightrag (20.240.61.67)

  1. SSH to Azure VM: ssh -i ~/.ssh/azure_alai alai-admin@20.240.61.67
  2. docker compose stop lightrag neo4j — graceful shutdown (~30s downtime)
  3. docker run alpine tar czf dumps each volume on VM
  4. docker compose start neo4j lightrag — resume
  5. shasum -a 256 *.tar.gz > MANIFEST.sha256 on VM
  6. Write README.md with restore procedure on VM
  7. SCP from VM to Mac Studio — download snapshot to ~/system/backups/lightrag/ (safety net)
  8. Azure offsite upload — Cool tier blob plockfrontstaging/lightrag-backup/<TS>/
  9. Azure rotation — keep last 8 snapshots (longer offsite retention)
  10. Local rotation — keep last 4 snapshots in ~/system/backups/lightrag/ (7-day safety, then deletable)

Downtime: ~60–90s every Sunday 04:00 (cloud LightRAG unavailable during backup).

Key change: Local Docker volumes are NO LONGER the source of truth. Azure VM volumes are primary. Local backups are now safety net only.

Why NOT docker compose pause

pause freezes LightRAG's async event loop. On unpause, uvicorn stays "running" but HTTP handler doesn't service new requests (container reports unhealthy). Requires full container restart to recover. The backup on 2026-04-18 hit this — backup itself was fine (volumes at rest during pause), but container needed restart afterwards. Switched to stop/start for future runs.

Azure storage details

Restore procedure

Restore to Azure VM (primary, production)

# On Mac Studio: pick snapshot
SNAPSHOT=~/system/backups/lightrag/20260418-085317
cd "$SNAPSHOT"
shasum -a 256 -c MANIFEST.sha256 || { echo "checksum mismatch, abort"; exit 1; }

# SCP to Azure VM
scp -i ~/.ssh/azure_alai -r "$SNAPSHOT" alai-admin@20.240.61.67:/tmp/restore/

# SSH to Azure VM
ssh -i ~/.ssh/azure_alai alai-admin@20.240.61.67

# On Azure VM:
cd /tmp/restore/$(basename "$SNAPSHOT")
shasum -a 256 -c MANIFEST.sha256 || { echo "checksum mismatch, abort"; exit 1; }

cd ~/lightrag
docker compose down

for vol in lightrag-data lightrag-kg lightrag-cache lightrag-neo4j-data; do
  docker volume rm $vol || true
  docker volume create $vol
  docker run --rm -v $vol:/dst -v /tmp/restore/$(basename "$SNAPSHOT"):/src alpine tar xzf /src/${vol}.tar.gz -C /dst
done

docker compose up -d

# Verify
curl http://localhost:9621/health
# From Mac Studio:
curl https://lightrag.basicconsulting.no/health

Restore to Mac Studio (rollback/emergency only)

Use case: Azure VM failure, need to restore local LightRAG as emergency fallback.

cd ~/system/docker/lightrag
docker compose down

# Pick a snapshot (local or download from Azure first)
SNAPSHOT=~/system/backups/lightrag/20260418-085317
cd "$SNAPSHOT"
shasum -a 256 -c MANIFEST.sha256 || { echo "checksum mismatch, abort"; exit 1; }

for vol in lightrag-data lightrag-kg lightrag-cache lightrag-neo4j-data; do
  docker volume rm $vol || true
  docker volume create $vol
  docker run --rm -v $vol:/dst -v "$SNAPSHOT":/src alpine tar xzf /src/${vol}.tar.gz -C /dst
done

cd ~/system/docker/lightrag
docker compose up -d

# Verify
curl http://localhost:9621/health

# IMPORTANT: Update consumer files to use localhost:9621 instead of cloud endpoint
# (see azure-lightrag-migration.md rollback procedure)

Azure Blob restore (download offsite backup)

Use case: Local backups lost, need to restore from Azure Blob offsite storage.

source ~/system/config/azure-lightrag-backup.env
TS=20260418-085317
RESTORE_DIR=~/system/backups/lightrag/azure-restore-$TS
mkdir -p "$RESTORE_DIR"
az storage blob download-batch \
  --account-name $AZURE_STORAGE_ACCOUNT \
  --account-key "$AZURE_STORAGE_KEY" \
  --source $AZURE_STORAGE_CONTAINER \
  --destination "$RESTORE_DIR" \
  --pattern "$TS/*"

# Verify checksums
cd "$RESTORE_DIR/$TS"
shasum -a 256 -c MANIFEST.sha256

# Then follow "Restore to Azure VM" or "Restore to Mac Studio" procedure above

Monitoring

Manual run

bash ~/system/tools/lightrag-backup.sh

Same 60–90s downtime applies. Log goes to same file.

Note: Post-migration (2026-04-18), script must be updated to SSH to Azure VM instead of using local Docker. See script comments for SSH-based backup procedure.

Document Owner: Skillforge
Last Updated: 2026-08-19 (MC #107355 local recovery and backup restart hardening)
Validated By: Kelsey Hightower (FlowForge), Martin Kleppmann (CodeCraft — data consistency)

Azure LightRAG Migration — Complete Runbook

Azure LightRAG Migration — Complete Runbook

CURRENT RUNTIME — AZURE ARCHITECTURE SUPERSEDED (MC #106747; G4 offline refresh MC #107082): LightRAG now runs on the local Mac Studio Docker stack. The last qualified read-only runtime evidence confirms vm-alai-lightrag is VM deallocated and the local lightrag/lightrag-neo4j topology is authoritative under /Users/makinja/system/lightrag-local/docker-compose.local.yml. MC #107082 prepared only an offline d38f399... client / 508e618... server G4 bundle; it created no authority receipt, UTC window, deploy, restart, or runtime effect. The Azure instructions below remain historical architecture and rollback context and are intentionally reported as documentation drift, not current execution guidance. See MC #106747/#106767, /Users/makinja/system/lightrag-local/mc106747-migration-note.md, and rag-queue-recovery-106978.md.

Domain note (2026-05-17): This doc was written when hostnames were lightrag.basicconsulting.no and ollama.basicconsulting.no. Both have since migrated to lightrag.alai.no and ollama.alai.no. Historical command examples below retain original hostnames for accuracy; use alai.no equivalents in live ops.

Status: COMPLETED 2026-04-18
Team Lead: Kelsey Hightower (FlowForge)
Architect: Petter Graff (CodeCraft)
Data Lead: Martin Kleppmann (CodeCraft)
Validator: Angie Jones (Proveo)
Documentation: Skillforge

Operational Update — 2026-05-20

MC #101607 repaired a query-path regression after Azure LightRAG health was green but /query returned HTTP 500 because LLM_MODEL=qwen3:8b-q8_0 was not available behind the Ollama tunnel. Azure /home/alai-admin/lightrag/.env now uses LLM_MODEL=llama3.1:8b, and the lightrag container was force-recreated without deleting volumes. Direct Azure endpoint http://20.240.61.67:9621 verified: /health healthy and /query HTTP 200. Local Anvil/Pi mock config currently uses the Azure direct URL as lightrag.base_url.

MC #101611 added client-side Cloudflare Access service-token support for the LightRAG wrapper and key consumers. When lightrag.base_url=https://lightrag.alai.no, provide LIGHTRAG_CF_ACCESS_CLIENT_ID and LIGHTRAG_CF_ACCESS_CLIENT_SECRET (or generic CF_ACCESS_CLIENT_ID / CF_ACCESS_CLIENT_SECRET). Do not commit secrets. Do not switch canonical lightrag.base_url from Azure direct to the public URL until a real service token validates /health and /query through Cloudflare Access.

Evidence: /tmp/verify-101607/SUMMARY.md, /tmp/verify-101609/SUMMARY.md, /tmp/verify-101611/SUMMARY.md.

Operational Update — 2026-07-28 (MC #101501)

~/system/tools/lightrag-bulk-upload.js ("the pump") default --rate increased from 10 to 25 docs/min. No launchd/cron caller passes --rate explicitly, so the hardcoded default (main(), previously parseInt(getArg('--rate') || '10')) was the effective live rate limit — it now reads || '25'. Doc-comment usage examples at the top of the file updated to match. Evidence: /tmp/evidence-101501/verification.txt (node -c syntax check + git diff).

CRITICAL Security Update — 2026-06-18/19

MC #103912: Public exposure incident and migration to app-layer JWT authentication.

Root Cause (2026-06-18)

Boot.sh/discover.js reported "LightRAG DOWN" (false negative). NSG vm-alai-lightragNSG only allows port 9621 from 46.46.240.0/20 + Cloudflare ranges; orchestrator host egress IP (92.221.168.61) was not whitelisted. LightRAG container was healthy throughout.

Security Incident Timeline

Why Cloudflare Access Failed:

Attempted 3 times:

  1. Wrong AUD tag (used app ID instead of correct AUD 45433679774e5bb11a3a5c284cf3a71e9f8865c93c513f5cc204e382e96cff8d)
  2. Wrong team name (alai instead of alai-no)
  3. originRequest.access enforces user JWT tokens (browser login), NOT service token headers

KEY LESSON: Verify allowlist/auth ONLY from a vantage NOT in any bypass list. CF Access IP-Bypass policy whitelists 92.221.168.61; testing from that IP always bypassed Access regardless of config.

Final Solution: App-Layer JWT Authentication

Enforcement Point: LightRAG FastAPI application (sbnb/lightrag container), NOT Cloudflare edge.

Configuration:

VM path: /home/alai-admin/lightrag/.env

AUTH_ACCOUNTS=alai-system:<password_hash>
TOKEN_SECRET=<jwt_secret>
WHITELIST_PATHS=/health

Credentials: Vaultwarden item 67dc69b5-b1cb-4892-970e-b4d60380378f (LightRAG API Auth).

Auth Flow:

  1. Client POSTs to /login with username + password (form data)
  2. Server validates credentials, returns JWT (48h expiry)
  3. Client includes Authorization: Bearer <token> on subsequent requests
  4. /health is PUBLIC (whitelisted for boot.sh probes)
  5. /query, /insert, /api/* require valid JWT (HTTP 401 without auth)

Verification (external-vantage required):

# Unauthenticated MUST return 401
curl -i -X POST https://lightrag.alai.no/query \
  -H 'Content-Type: application/json' -d '{"query":"test"}'
# Expected: HTTP/1.1 401 Unauthorized

# Health endpoint PUBLIC
curl -s https://lightrag.alai.no/health | jq -r '.status'
# Expected: "healthy"

# Authenticated access
TOKEN=$(curl -s -X POST 'https://lightrag.alai.no/login' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d 'username=alai-system&password=<from_vaultwarden>' \
  | jq -r '.access_token')

curl -s -X POST https://lightrag.alai.no/query \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"query":"test","mode":"naive","top_k":1}' | jq -r '.response'
# Expected: HTTP 200 + query results

Architecture:

External Client (unauthenticated)
  ↓ HTTPS POST /query
Cloudflare (lightrag.alai.no)
  ↓ Tunnel (c79ffe4d)
Azure VM cloudflared
  ↓ HTTP localhost:9621
LightRAG FastAPI Server
  → JWT Validator Middleware
  → HTTP 401 Unauthorized ❌ BLOCKED

External Client (authenticated with JWT)
  ↓ HTTPS POST /query + Authorization: Bearer <jwt>
Cloudflare → Tunnel → VM cloudflared → localhost:9621
LightRAG FastAPI Server
  → JWT validates (TOKEN_SECRET, 48h expiry)
  → HTTP 200 + query results ✅ ALLOWED

Known Residuals

Rotating LightRAG Credentials

# Generate new password hash
NEW_PASS=$(openssl rand -hex 32)
HASH=$(echo -n "$NEW_PASS" | sha256sum | awk '{print $1}')

# Update on VM
ssh -i ~/.ssh/azure_alai alai-admin@20.240.61.67
cd ~/lightrag
# Edit .env: AUTH_ACCOUNTS=alai-system:<new_hash>
docker compose restart lightrag

# Test
curl -s -X POST 'https://lightrag.alai.no/login' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d "username=alai-system&password=$NEW_PASS" | jq -r '.access_token'

# Update Vaultwarden item 67dc69b5-b1cb-4892-970e-b4d60380378f with new password

boot.sh / discover.js Integration

Config: ~/system/tools/alai-config-mock.json

{
  "lightrag.base_url": "https://lightrag.alai.no"
}

Credentials loaded from environment (NOT committed to git):

export LIGHTRAG_USERNAME="alai-system"
export LIGHTRAG_PASSWORD="<from_vaultwarden>"

boot.sh/discover.js authenticate on first request, cache JWT for 48h, refresh when expired.

Evidence: /tmp/evidence-103912/app-layer-auth-proof.md, /tmp/evidence-103912/verification-final.md

Executive Summary

Why migrated: Docker Desktop failed 3 times on 2026-04-18, causing LightRAG outages impacting all ALAI knowledge operations (discover.js, autocoder.js, retrieval-orchestrator.js). Local dependency became unacceptable single point of failure.

What changed:

Result:

Architecture Diagram

sequenceDiagram
    participant Consumer as Mac Studio Consumer<br/>(discover.js, autocoder.js, etc.)
    participant CF1 as Cloudflare<br/>lightrag.basicconsulting.no
    participant Tunnel1 as Mac Studio<br/>cloudflared tunnel
    participant VM as Azure VM<br/>20.240.61.67:9621<br/>LightRAG container
    participant CF2 as Cloudflare<br/>ollama.basicconsulting.no
    participant Tunnel2 as Mac Studio<br/>cloudflared tunnel
    participant Ollama as FORGE<br/>10.0.0.2:11434<br/>Ollama service

    Consumer->>CF1: HTTPS query request
    CF1->>Tunnel1: Route via tunnel
    Tunnel1->>VM: Forward to Azure VM:9621
    VM->>CF2: HTTPS request for LLM inference
    CF2->>Tunnel2: Route via tunnel (Zero Trust IP check)
    Tunnel2->>Ollama: Forward to FORGE:11434
    Ollama-->>Tunnel2: Inference response
    Tunnel2-->>CF2: Return
    CF2-->>VM: LLM result
    VM->>VM: Build knowledge graph (Neo4j)
    VM-->>Tunnel1: Query response
    Tunnel1-->>CF1: Return
    CF1-->>Consumer: HTTPS response

Key insight: Cloud LightRAG talks back to on-prem Ollama via second tunnel. Both services behind Cloudflare Zero Trust.

Resources Created

Azure Resource Group

Virtual Machine

Network Security Group (NSG)

Name: vm-alai-lightragNSG

Rule Name Priority Direction Port Source Purpose
default-allow-ssh 1000 Inbound 22 46.46.251.40/32 SSH from Mac Studio ISP
allow-lightrag-macstudio 100 Inbound 9621 46.46.251.40/32 Direct access (backup)
allow-cloudflare-lightrag 110 Inbound 9621 Cloudflare IP ranges Tunnel ingress

Important: Mac Studio ISP IP (46.46.251.40) is residential and may rotate. When rotation happens, SSH and direct API access will fail. Update NSG rules accordingly (see Troubleshooting).

Cloudflare DNS Records

Data Migration

Source

Restore Process

  1. Download from Azure Blob to VM /tmp/restore/
  2. shasum -a 256 -c MANIFEST.sha256 — verified all 4 tarballs
  3. Created 4 Docker volumes
  4. Extracted each tarball into its volume using Alpine container
  5. Started LightRAG + Neo4j containers
  6. Verified entity count in Neo4j matched pre-migration snapshot

Data loss: ZERO. Snapshot taken immediately before migration.

Consumer Files Cut Over

8 files updated from http://localhost:9621 → https://lightrag.basicconsulting.no:

File Purpose
~/system/tools/discover.js Universal search (tools, agents, docs, RAG)
~/system/tools/lightrag.js LightRAG client wrapper
~/system/tools/autocoder.js Code generation with RAG context
~/system/tools/lightrag-bulk-upload.js Batch document ingestion
~/system/tools/lightrag-migrate.js Migration utility
~/system/tools/lightrag-outbox-ingest.js Outbox processor
~/system/tools/retrieval-orchestrator.js Multi-source retrieval coordinator
~/system/tools/system-regression.sh Health check suite

Pre-cutover backups: Not created (files tracked in Git, easy revert via git restore).

Operational Procedures

Daily Health Check

# From Mac Studio
curl https://lightrag.basicconsulting.no/health

# Expected response:
# {"status":"healthy","working_directory":"/app/data", ...}

SSH to VM

ssh -i ~/.ssh/azure_alai alai-admin@20.240.61.67

Docker Container Management

# On Azure VM
cd ~/lightrag
docker compose ps              # Check status
docker compose logs -f         # Tail logs
docker compose restart         # Restart services
docker compose down && docker compose up -d  # Full restart

Health Check from VM (tests Ollama tunnel)

# On Azure VM
curl -s https://ollama.basicconsulting.no/api/tags | jq '.models | length'
# Should return model count (e.g., 12)

Azure Cost Check

# From Mac Studio
az consumption usage list --start-date $(date -u -v-7d +%Y-%m-%d) --end-date $(date -u +%Y-%m-%d) -o table

Stop LightRAG (for maintenance)

# On Azure VM
cd ~/lightrag
docker compose stop
# Restart after maintenance
docker compose start

Rollback Procedure — CRITICAL

When to rollback:

Expected rollback time: 5-15 minutes
Data loss risk: ZERO (local volumes preserved 7 days post-cutover)

Step 1: Revert Consumer URLs

# On Mac Studio
cd ~/system/tools
for file in discover.js lightrag.js autocoder.js lightrag-bulk-upload.js \
            lightrag-migrate.js lightrag-outbox-ingest.js \
            retrieval-orchestrator.js system-regression.sh; do
  sed -i '' 's|https://lightrag.basicconsulting.no|http://localhost:9621|g' "$file"
done

# Verify
grep -l "localhost:9621" *.js *.sh
# Should list all 8 files

Step 2: Restart Local Docker LightRAG

cd ~/system/docker/lightrag
docker compose up -d

# Wait for healthy status (30-60s)
docker compose ps

Step 3: Verify Local Service

curl http://localhost:9621/health
# Expected: {"status":"healthy", ...}

# Run regression suite
bash ~/system/tools/system-regression.sh
# LightRAG checks should PASS

Step 4: Deprovision Azure VM (optional, when convenient)

az group delete --name rg-alai-lightrag --yes --no-wait
# Deletes VM, NSG, disks, public IP
# Cloudflare DNS records remain (harmless)

Post-rollback actions:

  1. Update ~/system/docs/runbooks/lightrag-backup.md to revert to local backup flow
  2. Notify Alem in Slack #ops-alai
  3. Create MC task for post-mortem

Troubleshooting

Issue: "model not found" errors in LightRAG logs

Cause: Cloudflare tunnel ollama.basicconsulting.no routing to wrong backend.

Diagnosis:

# On Azure VM
curl -s https://ollama.basicconsulting.no/api/tags | jq '.models[].name'
# Should list qwen2.5-coder:32b-instruct-q8_0, bge-m3:latest, etc.

Fix:

  1. Check Mac Studio tunnel config: cat ~/.cloudflared/config.yml | grep -A2 ollama
  2. Should be service: http://10.0.0.2:11434 (FORGE), NOT http://localhost:11434 (ANVIL)
  3. If wrong: edit config, restart tunnel: launchctl kickstart -k gui/$(id -u)/com.john.cloudflared

Issue: SSH connection timeout

Cause: Mac Studio ISP rotated IP; NSG rule default-allow-ssh still has old IP (46.46.251.40/32).

Diagnosis:

# On Mac Studio
curl https://ifconfig.co
# Compare to NSG rule source IP
az network nsg rule show -g rg-alai-lightrag --nsg-name vm-alai-lightragNSG -n default-allow-ssh --query "sourceAddressPrefix"

Fix:

NEW_IP=$(curl -s https://ifconfig.co)
az network nsg rule update \
  -g rg-alai-lightrag \
  --nsg-name vm-alai-lightragNSG \
  -n default-allow-ssh \
  --source-address-prefixes "${NEW_IP}/32"

# Verify
ssh -i ~/.ssh/azure_alai alai-admin@20.240.61.67

Note: Use ifconfig.co or icanhazip.com, NOT ifconfig.me (returns CDN IP on some networks).

Issue: "connection refused" from consumer scripts

Cause: Cloudflare tunnel down or misconfigured.

Diagnosis:

# From Mac Studio
curl https://lightrag.basicconsulting.no/health
# If timeout/connection refused, tunnel is down

# Check tunnel process
ps aux | grep cloudflared
# Should show cloudflared running with config ~/.cloudflared/config.yml

# Check tunnel logs
tail -f ~/Library/Logs/cloudflared/cloudflared.log

Fix:

# Restart tunnel
launchctl kickstart -k gui/$(id -u)/com.john.cloudflared

# Wait 10-15s, retry
curl https://lightrag.basicconsulting.no/health

Issue: Slow query responses (>10s p95)

Cause 1: Network path issue (Mac Studio → Cloudflare → Azure → Cloudflare → Mac Studio).

Diagnosis:

# On Azure VM
time curl -s https://ollama.basicconsulting.no/api/tags > /dev/null
# Should be <100ms

# From Mac Studio
time curl -s https://lightrag.basicconsulting.no/health > /dev/null
# Should be <200ms

Cause 2: Ollama FORGE overloaded (other tasks using models).

Diagnosis:

# On Mac Studio
curl http://10.0.0.2:11434/api/ps
# Check running models

Fix: Identify and throttle/stop competing workloads on FORGE.

Issue: Neo4j "unable to allocate memory"

Cause: B2s_v2 has 8GB RAM; Neo4j + LightRAG + OS overhead can approach limit.

Diagnosis:

# On Azure VM
docker stats --no-stream
# Check memory usage percentages

free -h
# Check system memory

Fix (short-term):

# Restart containers to clear caches
cd ~/lightrag
docker compose restart

Fix (long-term): Upgrade to Standard_B2s_v2 → Standard_B4ms (4 vCPU, 16GB RAM, ~$60/month).

Cross-References

Validation Evidence

Completed by: Angie Jones (Proveo), 2026-04-18

Evidence bundle: ~/system/evidence/azure-lightrag-migration-20260418/SUMMARY.md

Next Steps

  1. Monitor for 7 days — track latency, cost, uptime. If stable, delete local Docker volumes.
  2. Update backup flow — migrate launchd plist to SSH-based Azure VM snapshot (see lightrag-azure-backup.md).
  3. Consider FORGE failover — expose FORGE Ollama via second tunnel endpoint for redundancy.
  4. Auto-scaling evaluation — if query volume grows, consider Azure Container Instances or AKS migration.

Document Owner: Skillforge
Last Updated: 2026-04-18
Approved By: Petter Graff (Architecture), Kelsey Hightower (Infra), Alem Basic (CEO)

Ollama Cloudflare Tunnel — Exposing Local Inference to Cloud

Ollama Cloudflare Tunnel — Exposing Local Inference to Cloud

Domain note (2026-05-17): This doc refers to ollama.basicconsulting.no — the legacy hostname. Current live endpoint: ollama.alai.no. Historical examples below retain original hostname for accuracy.

Owner: FlowForge (infra)
Implemented: 2026-04-18
Purpose: Expose Mac Studio Ollama (FORGE 10.0.0.2:11434) to Azure VM LightRAG via Cloudflare tunnel with Zero Trust IP whitelist

Why This Tunnel

Problem: LightRAG migrated to Azure VM to eliminate Docker Desktop single point of failure. But Ollama inference stays on Mac Studio (FORGE hardware, 40 local models including q8_0 quantizations).

Solution: Cloudflare tunnel from Mac Studio exposes ollama.basicconsulting.no → FORGE Ollama. Azure VM LightRAG calls this endpoint for LLM/embedding inference.

Trade-off:

Tunnel Configuration

Location: ~/.cloudflared/config.yml (Mac Studio)

tunnel: 3315a609-7934-45c5-ad0c-56d86d16374d
credentials-file: /Users/makinja/.cloudflared/3315a609-7934-45c5-ad0c-56d86d16374d.json

ingress:
  # ... other services ...
  
  - hostname: ollama.basicconsulting.no
    service: http://10.0.0.2:11434
  
  - service: http_status:404

Key points:

Restart tunnel after config changes:

launchctl kickstart -k gui/$(id -u)/com.john.cloudflared

Verify tunnel is up:

ps aux | grep cloudflared
curl https://ollama.basicconsulting.no/api/tags
# Should return JSON list of models

Zero Trust Policy — IP Whitelist

Policy Name: "Ollama Azure VM Only"
Application: ollama.basicconsulting.no
Type: Bypass (wildcard) + IP restrictions

Why bypass instead of strict Zero Trust auth:

Whitelisted IPs:

  1. Azure VM egress: Check current VM egress IP: ssh alai-admin@20.240.61.67 'curl -s https://ifconfig.co'
  2. Mac Studio (backup/testing): 46.46.251.40 (residential ISP, may rotate — see maintenance)

How policy works:

  1. Azure VM LightRAG makes HTTPS request to ollama.basicconsulting.no
  2. Cloudflare edge checks source IP against whitelist
  3. If match: forward to Mac Studio tunnel → FORGE Ollama
  4. If no match: return 403 Forbidden

Test from Azure VM:

ssh alai-admin@20.240.61.67
curl -s https://ollama.basicconsulting.no/api/tags | jq '.models | length'
# Should return model count (e.g., 12)

Test from random IP (should fail):

# From any non-whitelisted location
curl https://ollama.basicconsulting.no/api/tags
# Expected: 403 Forbidden or similar

IP Whitelist Maintenance

CRITICAL: Mac Studio ISP IP (46.46.251.40) is residential and WILL rotate periodically. When it does, both SSH to Azure VM and direct testing from Mac Studio will fail for Ollama tunnel testing.

Check Current Mac Studio IP

curl https://ifconfig.co
# Use ifconfig.co or icanhazip.com
# DO NOT use ifconfig.me (returns CDN IP on some networks)

Update NSG Rule (for Azure VM to access Mac Studio)

NEW_IP=$(curl -s https://ifconfig.co)

# Update SSH rule
az network nsg rule update \
  -g rg-alai-lightrag \
  --nsg-name vm-alai-lightragNSG \
  -n default-allow-ssh \
  --source-address-prefixes "${NEW_IP}/32"

# Update LightRAG access rule (if used for direct access)
az network nsg rule update \
  -g rg-alai-lightrag \
  --nsg-name vm-alai-lightragNSG \
  -n allow-lightrag-macstudio \
  --source-address-prefixes "${NEW_IP}/32"

Update Cloudflare Zero Trust Policy

Option 1: Cloudflare Dashboard

  1. Log in to Cloudflare Dashboard → Zero Trust
  2. Navigate to Access → Applications → "Ollama Azure VM Only"
  3. Edit policy → Update IP whitelist with new Mac Studio IP
  4. Save (takes effect within 30s)

Option 2: Cloudflare API (for automation)

# Get account ID and policy ID first (see Cloudflare API docs)
# Then use PATCH to update policy rules
# (Exact curl command omitted — requires API token with Zero Trust write access)

Verification after update:

curl https://ollama.basicconsulting.no/api/tags
# Should work from new Mac Studio IP

Failure Modes + Detection

Failure 1: Tunnel Process Down

Symptom: curl https://ollama.basicconsulting.no/api/tags returns connection timeout or 502 Bad Gateway.

Diagnosis:

ps aux | grep cloudflared
# If no process, tunnel is down

tail -f ~/Library/Logs/cloudflared/cloudflared.log
# Check for errors

Fix:

launchctl kickstart -k gui/$(id -u)/com.john.cloudflared
# Wait 10-15s
curl https://ollama.basicconsulting.no/api/tags

Persistent failure: Check launchd plist:

launchctl list | grep cloudflared
# Should show com.john.cloudflared

# If missing, reload plist
launchctl load ~/Library/LaunchAgents/com.john.cloudflared.plist

Failure 2: Model Not on Target Backend

Symptom: LightRAG logs show "model qwen2.5-coder:32b-instruct-q8_0 not found" or similar.

Diagnosis:

curl -s https://ollama.basicconsulting.no/api/tags | jq '.models[].name'
# Check which models are exposed

Cause: Tunnel points to wrong Ollama backend (ANVIL vs FORGE).

Fix:

# Check config
cat ~/.cloudflared/config.yml | grep -A2 "ollama.basicconsulting.no"

# Should be:
# service: http://10.0.0.2:11434  (FORGE)

# If wrong (e.g., http://localhost:11434 = ANVIL):
# Edit config, fix service URL
# Restart tunnel
launchctl kickstart -k gui/$(id -u)/com.john.cloudflared

Historical incident: 2026-04-18 mid-migration — tunnel initially pointed to ANVIL (localhost:11434). LightRAG couldn't find q8_0 models (only on FORGE). Changed to 10.0.0.2:11434, resolved immediately.

Failure 3: IP Whitelist Mismatch (403 Forbidden)

Symptom: Azure VM LightRAG logs show "403 Forbidden" or "Access denied" when calling Ollama endpoint.

Diagnosis:

# From Azure VM
ssh alai-admin@20.240.61.67
curl -v https://ollama.basicconsulting.no/api/tags 2>&1 | grep -E "HTTP|403"
# If 403, IP not whitelisted

# Check VM egress IP
curl -s https://ifconfig.co

Fix: Update Zero Trust policy (see IP Whitelist Maintenance section above).

Failure 4: Latency Spike (>500ms for /api/tags)

Symptom: Slow LightRAG responses; Ollama calls taking >1s for simple requests.

Diagnosis:

# From Azure VM
time curl -s https://ollama.basicconsulting.no/api/tags > /dev/null
# Should be 30-80ms typically

# From Mac Studio (local baseline)
time curl -s http://10.0.0.2:11434/api/tags > /dev/null
# Should be <10ms

Possible causes:

  1. Mac Studio network issue: Check Wi-Fi/Ethernet, router, ISP
  2. Cloudflare edge routing: Rare but possible; check Cloudflare status page
  3. FORGE overloaded: Other processes using Ollama heavily

Fix 3 (FORGE overload):

curl http://10.0.0.2:11434/api/ps
# Check running models and concurrent requests
# Identify and throttle/stop competing workloads

Performance Characteristics

Expected latency (Azure swedencentral ↔ Mac Studio Oslo):

Bandwidth: Not a bottleneck. Ollama API uses JSON over HTTPS; typical request/response <100KB except for large context prompts.

Throughput: Tunnel supports multiple concurrent requests. Bottleneck is FORGE hardware, not tunnel.

Cloudflare Tunnel SLA: 99.99% uptime (per Cloudflare SLA for paid plans). ALAI on Free plan but historically stable.

Security Considerations

Current model: IP whitelist via Cloudflare Zero Trust bypass policy.

Threat model:

Future hardening options:

  1. Service tokens: Replace IP whitelist with Cloudflare service token in request headers
  2. Mutual TLS: Require client cert from Azure VM
  3. VPN: Azure VNet peering to Mac Studio (complex, likely overkill)

Current assessment: IP whitelist sufficient for internal infrastructure. Service tokens recommended if IP rotation becomes operationally painful.

Monitoring

Health check (from Mac Studio):

curl https://ollama.basicconsulting.no/api/tags
# Should return model list

Health check (from Azure VM):

ssh alai-admin@20.240.61.67 'curl -s https://ollama.basicconsulting.no/api/tags | jq ".models | length"'
# Should return model count

Tunnel logs:

tail -f ~/Library/Logs/cloudflared/cloudflared.log

Cloudflare Analytics:

Rollback / Emergency Cutover

If tunnel becomes persistently unstable:

Option 1: Move LightRAG back to Mac Studio (see azure-lightrag-migration.md rollback procedure).

Option 2: Deploy Ollama to Azure (longer-term, requires GPU VM or accept slower inference on CPU):

  1. Provision Azure VM with GPU (e.g., Standard_NC4as_T4_v3, ~$500/month)
  2. Install Ollama on Azure VM
  3. Pull required models (qwen2.5-coder:32b-instruct-q8_0, bge-m3:latest)
  4. Update LightRAG .env: LLM_BINDING_HOST=http://localhost:11434
  5. Test inference latency (will be slower than FORGE M2 Ultra)

Option 3: Use Ollama Cloud / OpenAI API (cost implications, loses on-prem privacy):

Recommendation: Keep current tunnel setup unless persistent failures. FORGE uptime historically excellent.

Document Owner: Skillforge
Last Updated: 2026-04-18
Validated By: Kelsey Hightower (FlowForge), Parisa Tabriz (Securion — security review)

SENTINEL Reliability Sprint — System Overview

SENTINEL Reliability Sprint — System Overview

Status: COMPLETE — 2026-04-19 Sprint Leader: Petter Graff (L1) Team: Kelsey Hightower (DevOps), Martin Kleppmann (data/events), Angie Jones (validator), Skillforge (docs) Trigger: CEO complaint 2026-04-19 — "sistem pada, gubim novac, blind sam"

Executive Summary

Before this sprint: 16 dead daemons, 4 active public surface incidents (lumiscare 502, mc 502, snowit NXDOMAIN, bilko TLS mismatch), email intake dead 53 days, Slack alert bot SIGKILL'd. Zero automated alerts reached Alem for 15 of 17 incidents in 30-day window.

After this sprint: 12 dead daemons (4 fixed), 6 public surface monitors (BetterStack + ops-watchdog), email DLQ operational, Slack bot alive with email fallback, TLS cert expiry monitor, HiveMind alert subscribers.

Key metric: Time to alert on public surface down: was ∞ (never) → now ≤ 60 seconds (Slack + email).

Sprint Metrics (Tool-Verified)

Metric Before After Evidence
Dead daemons 16 12 launchctl list snapshot
Public surface monitors 1 (Drop only) 7 (6 new) BetterStack + ops-watchdog.json
Alert delivery channels 1 (email) 3 (Slack #ops + email + digest) Slack bot PID + email-fallback config
Email DLQ none ~/system/logs/email-dlq.jsonl File exists + tested with synthetic fail
Cert expiry monitoring none com.alai.cert-expiry-monitor launchctl list
HiveMind alert subscribers 0 2 (kind=alert, kind=intake) hivemind.db subscriptions table
Time to alert (public 502) ∞ (never) 60s (Slack) / 180s (BetterStack) Angie validation Task 6

Alert Flow Diagram

flowchart LR
    A[Event: Service Down] --> B{Detection}
    B -->|Internal| C[ops-watchdog]
    B -->|External| D[BetterStack]
    
    C --> E{Slack Bot Alive?}
    D --> F[Slack Webhook]
    
    E -->|Yes| G[Slack #ops]
    E -->|No| H[Email Fallback]
    F --> G
    
    G --> I[On-Call: John/Alem]
    H --> I
    
    J[Daily Digest] --> K[john-daily-digest]
    K --> L[Slack DM to Alem 08:00]
    
    style A fill:#ff6b6b
    style G fill:#51cf66
    style H fill:#ffd43b
    style I fill:#339af0

Alert Priority Routing:

Current Architecture After Sprint

1. Alert Channels (3 layers)

Channel Purpose Latency Target Config
Slack #ops Technical alerts (primary) ≤ 60s ~/system/config/ops-watchdog.json + BetterStack webhook
Email fallback When Slack bot down OR Slack API fails ≤ 90s ops-watchdog.json → email_fallback.enabled = true
john-daily-digest Summary layer (non-urgent) Daily 08:00 CET com.alai.john-daily-digest → Alem DM

Critical: Slack bot itself (com.john.slack-bot) is monitored by ops-watchdog. If messenger dies, email fallback activates automatically.

2. Monitoring Layers (2 independent)

Layer 1: BetterStack (External, SaaS)

Layer 2: ops-watchdog (Internal, Mac Studio)

Layer 3: TLS Cert Expiry (Scheduled Daily)

Layer 4: Cloudflared Tunnel Health (Critical SPOF)

What Was Fixed (Honest Accounting)

Phase 1: Revive Alert Messenger (COMPLETE)

Task 1a: Restart Slack bot

Task 1b: Add slack-bot to ops-watchdog critical list

Task 1c: Fix dead daemons

Phase 2: Public Surface Monitoring (COMPLETE)

Task 2a: BetterStack — 6 new monitors

Task 2b: ops-watchdog extended — public endpoint checks

Task 2c: TLS cert expiry monitor

Task 2d: Cloudflared tunnel health alert

Phase 3: Email Intake Revival (COMPLETE)

Task 3a: Vault ETIMEDOUT root cause

Task 3b: Dead-letter queue for email ingestion

Task 3c: Contact form intake documentation

Phase 4: HiveMind Event Bus Fixes (COMPLETE)

Task 4a: Subscribe dead event kinds

Task 4b: Evidence gate on task outcomes

What Was NOT Fixed (Honest)

Being direct — these are real gaps not covered by this sprint:

  1. alai.no contact form is dead stub — No backend action on form submission. Visitors think they're submitting but nothing happens. URGENT ticket #8379 created (owner: Vizu — frontend form + backend hook).

  2. snowit.ba DNS NXDOMAIN — Domain lapsed or DNS misconfigured. Owner decision needed: renew domain, redirect to alai.no, or sunset? MC ticket #8374 assigned to John.

  3. Mac Studio tunnel SPOF — All 26 cloudflared hostnames through one tunnel on one consumer machine. If Mac sleeps/crashes/loses power, ALL public surfaces die simultaneously. Phase 2 sprint (2-week scope, Azure secondary tunnel + cost optimization).

  4. 12 remaining dead daemons — Sprint fixed 4 of 16. Remaining 12: some are deprecated (com.john.unified-dispatcher), some need creds (com.john.b2-offsite-backup), some need investigation (com.alai.meta-agent-loop exit 78). Phase 2 sprint.

  5. Vaultwarden Docker down — Root cause of email intake death was vault container stopped on Azure VM. Why it stopped is unknown (no crash logs, VM uptime 47d). Needs monitoring: add vault.alai.no to Docker health check script.

  6. sign.alai.no redirect storm — 2388 cloudflared errors in 7-day log. Root cause unknown (Documenso redirect loop?). BetterStack now monitors it but fix requires Documenso investigation.

  7. b2-offsite-backup exit 1 — Possible B2 quota exceeded or creds issue. Sprint does not address backup verification. If backup is silently failing, data loss risk accumulates. Needs Backblaze billing review.

  8. Domain expiry monitoring — No whois check for snowit.ba, getdrop.no, alai.no. A lapsed domain = NXDOMAIN with zero alert until BetterStack fires HTTP error. Needs separate com.alai.domain-expiry-monitor daemon.

  9. VM-level monitoring — vm-alai-support hosts BookStack, Vault, Documenso. If the VM stops, all 3 go down. BetterStack HTTP monitors cover public URLs but not Azure VM health. Azure Monitor or SSH keepalive not in scope.

  10. HiveMind 33,406 unread events — Sprint fixes kind=alert and kind=intake subscribers. Other kinds (briefing, research, skill_proposal) remain with zero subscribers. Write-only archive.

Operations

How to Check System Health

# 1. Alert messenger alive
node ~/system/tools/slack.js send ops "sentinel health check"
# Should appear in #ops within 3 sec

# 2. ops-watchdog status
launchctl list | grep ops-watchdog
# Should show com.john.ops-watchdog with LastExit=0, non-zero PID

# 3. Dead daemon count
launchctl list | grep -E "alai|john" | awk '$2 != "0" && $1 !~ /^[0-9]+/' | wc -l
# Should be ≤ 12 (was 16 before sprint)

# 4. Email DLQ size
wc -l ~/system/logs/email-dlq.jsonl
# Should be 0-2 entries (if > 5, investigate vault health)

# 5. Cert expiry next run
launchctl list | grep cert-expiry
# Should show com.alai.cert-expiry-monitor with LastExit=0

# 6. BetterStack coverage (manual)
# Open https://betterstack.com/uptime (login: alem@alai.no)
# Verify 7 monitors green (Drop + 6 ALAI endpoints)

# 7. Public surface live check
for url in https://alai.no https://lumiscare.alai.no https://getdrop.no https://docs.alai.no https://vault.alai.no https://sign.alai.no; do
  echo -n "$url: "
  curl -sfL --max-time 10 -o /dev/null -w '%{http_code}\n' "$url"
done
# All should return 200 or 3xx (except snowit.ba NXDOMAIN)

How to Add New Endpoint to Monitor

BetterStack (3-min external check):

  1. Log into https://betterstack.com/uptime (alem@alai.no)
  2. Click Monitors → Create Monitor
  3. Fill: Name, URL, Interval (3 min), Expected Status (200), Keyword check (optional)
  4. Select Escalation Policy: "Drop Production Incidents" (routes to #ops)
  5. Save

ops-watchdog (2-min internal check):

  1. Edit ~/system/config/ops-watchdog.json
  2. Add entry to custom_health_checks: 
    "public-newservice": {
  "description": "newservice.alai.no",
  "check_command": "curl -sf --max-time 10 https://newservice.alai.no/ | grep -q 'Expected Text'",
  "alert_message": "⚠️ PUBLIC SURFACE DOWN: newservice.alai.no unreachable",
  "consecutive_failures_required": 2
}
  3. Restart ops-watchdog: launchctl kickstart -k gui/$(id -u)/com.john.ops-watchdog
  4. Test: Stop service, wait 4 min (2 cycles), verify alert in #ops

How to Restart Key Daemons Safely

# Slack bot (alert messenger)
launchctl kickstart -k gui/$(id -u)/com.john.slack-bot
# Verify: node ~/system/tools/slack.js send ops "test after restart"

# ops-watchdog (monitoring daemon)
launchctl kickstart -k gui/$(id -u)/com.john.ops-watchdog
# Verify: tail -f ~/system/logs/ops-watchdog.log (should show "Starting check cycle...")

# Email agent (email intake)
launchctl kickstart -k gui/$(id -u)/com.john.email-agent
# Verify: test -f /tmp/email-agent-last-success && echo "OK"

# Cloudflared tunnel (ALL 26 public hostnames)
# DANGER: This takes down ALL public surfaces for 3-5 seconds
launchctl kickstart -k gui/$(id -u)/com.john.cloudflared
# Verify: curl -sf https://alai.no (should return 200 within 10s)

# MC Dashboard (internal UI)
launchctl kickstart -k gui/$(id -u)/com.john.mc-dashboard
# Verify: curl -sf http://localhost:3030 | grep -q 'Mission Control'

Cross-References

Evidence bundle:

Success Criteria (CEO-Reportable)

After this sprint, the following are TRUE (tool-verified):

✅ 4 active incidents found during audit RESOLVED or ticketed (lumiscare 502 → ticket #8373, mc 502 → fixed, snowit NXDOMAIN → ticket #8374, bilko TLS → ticket #8375)

✅ Alem receives Slack alert ≤ 60s of any of 6 public surfaces going down (validated: stopped cloudflared, alert arrived in 47s via email fallback + 53s via Slack after bot restart)

✅ Email intake pipeline alive (vault restarted, bw unlock succeeds, email-agent LastExit=0)

✅ DLQ operational (tested: broke bw, sent email, envelope landed in DLQ, replayed successfully)

✅ TLS cert expiry caught ≥ 30 days before lapse (com.alai.cert-expiry-monitor runs daily 07:00, alerts at 30/14/7 days)

✅ Dead daemon count 16 → 12 (4 fixed: forge-watchdog, health-monitor, mc-dashboard, john-daily-digest)

✅ HiveMind alert + intake kinds have live subscribers (2 subscribers registered, smoke test passed)

One-Liner Summary (for Alem)

Već imamo watchdogs, BetterStack, i ops-watchdog — ali Slack bot (poštar) je bio SIGKILL-ovan pa je sve bilo tiho; email intake mrtav 53 dana; 4 public endpointa pala RIGHT NOW a niko te nije obavijestio. Ovaj sprint je popravio poštara, dodao 6 BetterStack monitora, napravio DLQ za email, i sada dobijaš Slack alert za 60 sekundi ako bilo koji public surface padne. 16 dead daemona → 12 (4 fixed). Phase 2 sprint dolazi za secondary tunnel + 12 preostalih daemona.

Sprint completed: 2026-04-19 10:24 CET
Validation: Angie Jones (Task 6) — E2E evidence at ~/system/evidence/sentinel-sprint-2026-04-19/SUMMARY.md
Documentation: Skillforge (Task 7) — This runbook + 2 companion docs

Incident Response Playbook

Incident Response Playbook

Purpose: When an alert fires, what to do immediately. No research, no debugging — just triage → diagnose → escalate/fix.
Audience: John (primary), Alem (fallback), FlowForge/CodeCraft agents (delegated fixes)
Last updated: 2026-04-19 (SENTINEL Sprint)

Alert Triage Matrix

When you see this alert → do this immediately:

Alert Message Severity First Action Diagnostic Commands Escalate If
"⚠️ PUBLIC SURFACE DOWN: alai.no" P0 Verify tunnel + origin curl -I https://alai.no
launchctl list | grep cloudflared
tail -50 ~/Library/Logs/ALAI/cloudflared-error.log		 Down > 5 min → Alem directly
"⚠️ PUBLIC SURFACE DOWN: lumiscare.alai.no" P0 Check Docker containers docker ps | grep lumiscare
docker logs lumiscare-web
curl http://localhost:4001		 Container stopped → restart, if fail → Alem
"⚠️ PUBLIC SURFACE DOWN: getdrop.no" P0 Check Vercel deployment curl -I https://getdrop.no
vercel ls drop-landing
Vercel dashboard		 Vercel outage or DNS → Alem
"⚠️ PUBLIC SURFACE DOWN: docs/vault/sign.alai.no" P0 Check Azure VM + Docker ssh alai-admin@4.223.110.181
docker ps
systemctl status docker		 VM down or out of disk → Alem
"⚠️ PUBLIC SURFACE DOWN: snowit.ba" P1 Check DNS + domain expiry dig snowit.ba
whois snowit.ba | grep -i expiry		 Domain lapsed → Alem (billing decision)
"[SENTINEL ALERT] ops-watchdog" P1 Check which service died launchctl list | grep -E "alai|john"
View plist logs: tail -50 ~/Library/Logs/ALAI/<service>.log		 Critical service down > 10 min → escalate
"Slack bot DOWN — email fallback active" P0 Restart slack-bot launchctl kickstart -k gui/$(id -u)/com.john.slack-bot
node ~/system/tools/slack.js send ops "test after restart"		 Restart fails → Alem (all alerts via email until fixed)
"Email DLQ size > 5 entries" P1 Check vault + bw CLI bw unlock --check
curl -I https://vault.alai.no
wc -l ~/system/logs/email-dlq.jsonl		 Vault down > 1 hr OR DLQ > 20 → Alem
"TLS cert expiry: in 7 days" P1 Verify cert date + renew echo | openssl s_client -connect <domain>:443 -servername <domain> 2>/dev/null | openssl x509 -noout -enddate
Cloudflare dashboard → SSL/TLS		 Cert renew fails → Alem (public outage risk)
"[HM-ALERT] agent: " P2 Check HiveMind source sqlite3 ~/system/databases/hivemind.db "SELECT * FROM events WHERE kind='alert' ORDER BY timestamp DESC LIMIT 5" Agent loop detected OR repeated fail → investigate
"[INTAKE] source: " P2 Review MC task auto-created node ~/system/tools/mc.js list --status pending
Check intake source (email/form/Slack)		 Spam OR malformed intake → tune classification
"[NO-EVIDENCE] Task # done" P3 Check sidecar + re-validate tail ~/system/logs/task-outcomes-pending-evidence.jsonl
node ~/system/tools/mc.js show <id>		 Builder repeatedly skips evidence → Proveo re-validation

Common Incidents (From 30-Day Ledger)

1. Drop Landing Page 502 (Happened: Apr 7, 9)

Symptoms: BetterStack alert "Drop Landing Page DOWN" (HTTP 502 or DNS timeout)

Diagnosis:

# 1. Check Vercel deployment status
curl -I https://getdrop.no
vercel ls drop-landing

# 2. Check DNS
dig getdrop.no

# 3. Check Vercel dashboard
# Open: https://vercel.com/basic-as/drop-landing
# Look for: "Deployment Failed" or "Domain Configuration Error"

Fix:

Escalate if: Down > 10 min AND revenue event (customer trying to pay) → Alem directly via phone +47 404 74 251

Post-incident: Update Drop incident log at ~/system/evidence/drop-incidents.md

2. LumisCare 502 (Happened: Apr 19 — silent for hours)

Symptoms: "⚠️ PUBLIC SURFACE DOWN: lumiscare.alai.no" (HTTP 502 — connection refused :4001)

Diagnosis:

# 1. Check Docker containers
docker ps | grep lumiscare
# Expected: lumiscare-web (port 4001), lumiscare-api (port 8090), lumiscare-ollama (port 4003)

# 2. If missing, check stopped containers
docker ps -a | grep lumiscare

# 3. Check logs
docker logs lumiscare-web --tail 50
docker logs lumiscare-api --tail 50

Fix:

# If containers stopped, restart
cd ~/projects/lumiscare
docker compose up -d

# Verify
curl -I http://localhost:4001
curl -I http://localhost:8090

# Check cloudflared tunnel routing
curl -I https://lumiscare.alai.no

Escalate if: Container restart fails with error OR OOM killed repeatedly → Alem (may need Azure migration for LumisCare)

Root cause notes: LumisCare Docker containers were stopped on Apr 19 for unknown reason (no crash logs, Mac uptime 47d). Possibly manual docker stop or OOM. Needs Docker health check monitoring.

3. Slack Bot SIGKILL (Happened: unknown date — killed ALL alerts)

Symptoms: No alerts in #ops for days, launchctl shows com.john.slack-bot with exit -9, email fallback activates

Diagnosis:

# 1. Check if bot is dead
launchctl list | grep slack-bot
# If PID = "-" and Status = "-9" → killed

# 2. Check memory usage history (if available)
# OOM kill leaves no direct trace, but check system.log
log show --predicate 'eventMessage contains "slack-bot"' --info --last 1h

# 3. Test Slack API reachability
curl -I https://slack.com/api/api.test

Fix:

# 1. Restart bot
launchctl kickstart -k gui/$(id -u)/com.john.slack-bot

# 2. Verify alive
launchctl list | grep slack-bot
# Should show non-zero PID, LastExit = 0

# 3. Test alert delivery
node ~/system/tools/slack.js send ops "sentinel: slack-bot restarted after SIGKILL"

# 4. Check if alert appears in #ops within 5 sec

Escalate if: Restart fails OR bot dies again within 1 hour → Alem (memory leak investigation needed, may need rewrite)

Prevention: After sprint, ops-watchdog monitors slack-bot itself. If bot dies, email fallback activates automatically.

4. Email Intake Pipeline Dead (Happened: Feb 25 — silent 53 days)

Symptoms: "Email DLQ size > 5 entries" OR manual discovery (email-agent.log not updated in days)

Diagnosis:

# 1. Check email-agent daemon
launchctl list | grep email-agent
# If LastExit != 0 → daemon crashed

# 2. Check vault connectivity
bw unlock --check
# If fails → vault session expired or Vaultwarden down

# 3. Check Vaultwarden Docker (Azure VM)
ssh alai-admin@4.223.110.181
docker ps | grep vaultwarden
# If missing → container stopped

# 4. Check DLQ size
wc -l ~/system/logs/email-dlq.jsonl

Fix:

# If vault session expired (ETIMEDOUT):
# 1. Restart Vaultwarden on Azure VM
ssh alai-admin@4.223.110.181 "cd ~/docker/vaultwarden && docker compose up -d"

# 2. Unlock vault locally
bw unlock
# Enter master password (from Alem or ~/system/config/.vault-session if cached)

# 3. Restart email-agent
launchctl kickstart -k gui/$(id -u)/com.john.email-agent

# 4. Replay DLQ
bash ~/system/tools/email-dlq-replay.sh

# 5. Verify DLQ cleared
wc -l ~/system/logs/email-dlq.jsonl
# Should be 0 or 1

Escalate if: Vaultwarden container won't start OR bw unlock fails with password error → Alem (may need Bitwarden master password reset)

Prevention: After sprint, email-agent writes failed emails to DLQ. Alert fires if DLQ > 5 entries. Vault downtime no longer causes silent email loss.

5. MC Dashboard 502 (Happened: Apr 19)

Symptoms: "⚠️ PUBLIC SURFACE DOWN: mc.alai.no" (HTTP 502 — connection refused :3030)

Diagnosis:

# 1. Check mc-dashboard daemon
launchctl list | grep mc-dashboard
# If LastExit = 1 → daemon crashed

# 2. Check local port
curl -I http://localhost:3030
# If connection refused → service not running

# 3. Check logs
tail -50 ~/system/logs/mc-dashboard.log

Fix:

# 1. Restart daemon
launchctl kickstart -k gui/$(id -u)/com.john.mc-dashboard

# 2. Verify local
curl -I http://localhost:3030
# Should return 200

# 3. Verify public (through cloudflared tunnel)
curl -I https://mc.alai.no

Escalate if: Restart fails with "missing node_modules" OR "port 3030 in use" → CodeCraft fix (dependency or port conflict issue)

6. Cloudflared Tunnel Down (SPOF — ALL 26 hostnames die)

Symptoms: Multiple BetterStack alerts simultaneously (alai.no + lumiscare.alai.no + docs + vault + sign + getdrop all down within 1 min)

Diagnosis:

# 1. Check cloudflared daemon
launchctl list | grep cloudflared
# If PID = "-" → tunnel dead

# 2. Check error log
tail -100 ~/Library/Logs/ALAI/cloudflared-error.log

# 3. Check Cloudflare Zero Trust dashboard
# Open: https://one.dash.cloudflare.com
# Navigate: Networks → Tunnels → "alai-main-tunnel"
# Look for: "Tunnel Disconnected" or "No Healthy Connectors"

Fix:

# 1. Restart tunnel
launchctl kickstart -k gui/$(id -u)/com.john.cloudflared

# 2. Wait 10 seconds for reconnect

# 3. Verify public endpoints
for url in https://alai.no https://lumiscare.alai.no https://getdrop.no; do
  echo -n "$url: "
  curl -sfL --max-time 10 -o /dev/null -w '%{http_code}\n' "$url"
done

Escalate if:

CRITICAL: This is the single biggest SPOF in ALAI infrastructure. Phase 2 sprint (deferred) will add secondary tunnel on Azure VM.

7. Azure VM SSH Timeout (Happened: Apr 19)

Symptoms: ssh alai-admin@4.223.110.181 hangs or "Connection timed out"

Diagnosis:

# 1. Check VM reachability
ping -c 3 4.223.110.181

# 2. Check Azure portal
# Open: https://portal.azure.com
# Navigate: Resource groups → alai-support → vm-alai-support
# Look for: "VM Status: Stopped" or "Networking issues"

# 3. Check NSG rules
# Azure portal → vm-alai-support → Networking → Inbound port rules
# Verify: Port 22 (SSH) is allowed from your IP

Fix:

Escalate if: VM won't start OR restart fails → Alem (Azure billing issue OR quota exceeded)

Impact: If vm-alai-support is down, these services die: BookStack (docs.alai.no), Vaultwarden (vault.alai.no), Documenso (sign.alai.no). BetterStack will fire 3 simultaneous alerts.

8. TLS Cert Expiry Warning (bilko-demo expires Jun 22, 2026)

Symptoms: "TLS cert expiry: bilko-demo.basicconsulting.no in 7 days" (alert fires 7 days before lapse)

Diagnosis:

# 1. Verify cert expiry date
echo | openssl s_client -connect bilko-demo.basicconsulting.no:443 -servername bilko-demo.basicconsulting.no 2>/dev/null | openssl x509 -noout -enddate

# 2. Check Cloudflare SSL settings
# Open: https://dash.cloudflare.com
# Select domain: basicconsulting.no
# Navigate: SSL/TLS → Edge Certificates
# Look for: "Universal SSL" status + expiry date

Fix:

Escalate if: Cloudflare renewal fails OR custom cert upload fails → Alem (public outage imminent within 7 days)

Escalation Path

Incident Type Escalate To When Contact Method
Public surface down > 5 min Alem Immediately Slack DM + Phone +47 404 74 251
Revenue event (Drop payment failing) Alem Immediately Phone first, Slack second
Security breach or suspicious activity Alem + Securion Immediately Slack #ops + Email alembasic@gmail.com
PI licenca revoked or legal issue Alem Within 1 hour Phone + Email
Azure VM / billing / quota issue Alem Within 30 min Slack + Email (needs Azure portal access)
Mac Studio hardware (power/network) Alem Immediately Phone (may need physical access)
Cloudflared tunnel down > 10 min Alem Immediately ALL public surfaces offline
Builder agent repeated failures (3+ in 1 hour) Petter Graff (specialist) Within 1 hour Slack #ops → delegate fix
Slack bot down (messenger dead) John (self-fix) Within 5 min Email fallback active, restart bot
Daemon down (non-critical) John (self-fix) Within 15 min Investigate + restart or ticket for agent

CRITICAL: If John (orchestrator) is offline, all P0 alerts route to Alem via email (alembasic@gmail.com). Check inbox every 15 min during incidents.

Runbook References

For step-by-step daemon restart procedures, see:

For safe daemon unload/reload:

# Unload (stop daemon, keep plist)
launchctl unload -w ~/Library/LaunchAgents/com.john.<service>.plist

# Load (start daemon from plist)
launchctl load -w ~/Library/LaunchAgents/com.john.<service>.plist

# Kickstart (restart without unload/load)
launchctl kickstart -k gui/$(id -u)/com.john.<service>

Playbook maintained by: Skillforge (SENTINEL Task 7)
Last incident review: 2026-04-19 (30-day ledger: 17 incidents, 2 with alerts, 15 silent)
Next review: After Phase 2 sprint (secondary tunnel + 12 dead daemons fixed)

Alert Routing — Channel Mapping & SLA

Alert Routing — Channel Mapping & SLA

Purpose: Who gets what alert, on which channel, with what latency target.
Audience: John (orchestrator), Alem (CEO), ops-watchdog daemon, agent builders
Last updated: 2026-04-19 (SENTINEL Sprint Task 7)

Alert Severity Table

Severity Channel Target Audience Latency SLA Retry Logic Example Alerts
P0 Critical Slack #ops + Email fallback Alem + John ≤ 60s Retry 3x, then email Public surface 502 (≥2 cycles), Cloudflared tunnel down, Slack bot SIGKILL
P1 High Slack #ops John (on-call) ≤ 3 min Retry 2x, then DLQ Daemon exit nonzero (critical services), Email DLQ > 5 entries, TLS cert expiry ≤ 7 days
P2 Info john-daily-digest Alem (morning review) Daily 08:00 CET Buffered, no retry New skill proposal, briefing summary, task ready for review, HiveMind research
P3 Debug Log file only Archive (no human) n/a Write once Heartbeat OK pulses, ops-watchdog check passed, daemon start/stop routine

Key principle: P0/P1 alerts MUST be actionable. If no action is needed → downgrade to P2 or P3. Alert fatigue = blind system.

Channel Routing Details

1. Slack #ops (Primary Technical Channel)

Purpose: Real-time technical alerts requiring immediate investigation or fix.

Routing sources:

Target audience:

Message format:

[SOURCE] Severity: Alert Title
Details: <brief description>
Time: 2026-04-19 10:24:15 CET
Runbook: ~/system/docs/runbooks/<name>.md (if available)

Example:

[SENTINEL ALERT] P0: ⚠️ PUBLIC SURFACE DOWN: alai.no
Details: HTTP 502 — connection refused (detected 2 consecutive cycles)
Time: 2026-04-19 10:24:15 CET
Runbook: ~/system/docs/runbooks/incident-response-playbook.md#1-alai-no-502

Cooldown: Same alert within 15 min = suppressed (prevents spam from flapping services). After 15 min silence, next occurrence fires new alert.

Alert count limit: If same service fires > 5 alerts in 1 hour → escalate to P0 + tag Alem ("Repeated failure — may need architectural fix").

2. Email Fallback (alembasic@gmail.com)

Purpose: Backup channel when Slack #ops is unreachable OR Slack bot (com.john.slack-bot) is dead.

Trigger conditions:

  1. Slack bot PID = "-" (daemon stopped/killed) — ops-watchdog detects this via critical_services check
  2. Slack API returns 5xx error for 3 consecutive attempts (Slack platform outage)
  3. Ops-watchdog config email_fallback.enabled = true (set after SENTINEL sprint)

Routing logic (ops-watchdog):

# Pseudocode from ops-watchdog daemon:
if slack_bot_dead() or slack_api_unavailable():
    send_email(
        to="alembasic@gmail.com",
        subject="[SENTINEL FALLBACK] Alert: <title>",
        body="Slack #ops unreachable. Alert details:\n<full alert message>"
    )

Latency SLA: ≤ 90s from alert trigger (Slack primary is 60s, email fallback is 30s slower due to SMTP handshake).

Example email:

Subject: [SENTINEL FALLBACK] P0: PUBLIC SURFACE DOWN: alai.no
Body:
Slack #ops is unreachable (slack-bot SIGKILL'd).
Alert routed via email fallback.

Alert: ⚠️ PUBLIC SURFACE DOWN: alai.no
Details: HTTP 502 — connection refused (detected 2 consecutive cycles)
Time: 2026-04-19 10:24:15 CET
Source: ops-watchdog (internal monitor)

Action: Restart cloudflared tunnel + verify origin
Runbook: ~/system/docs/runbooks/incident-response-playbook.md#1-alai-no-502

— ops-watchdog daemon

Alert count in fallback mode: All P0 alerts go to email. P1 alerts are buffered to DLQ (~/system/logs/alert-dlq.jsonl) until Slack bot is restored. After restoration, DLQ replays to Slack #ops.

3. john-daily-digest (Summary Layer)

Purpose: Non-urgent aggregated summary for Alem's morning review (08:00 CET).

Content sources:

Delivery:

Example digest:

Good morning Alem. Overnight summary (2026-04-18 18:00 → 2026-04-19 08:00 CET):

## Tasks Completed (3)
- #8370: SENTINEL T2a BetterStack 6 monitors (FlowForge) — 6 new public endpoint monitors added
- #8371: SENTINEL T3b Email DLQ (CodeCraft) — Dead-letter queue operational, tested with vault failure
- #8372: SENTINEL T7 BookStack 3 runbooks (Skillforge) — Documentation complete

## New Intake (2)
- Email from prospect (forwarded by John): Inquiring about AI consulting for retail chain (200 stores)
- Slack message from partner: Entur wants to schedule follow-up call for RAG demo

## Cost Alert (1)
- Yesterday spend: 67 USD (above 50 USD threshold)
  - Azure VM: 22 USD
  - OpenAI API: 38 USD (Opus 4 tasks)
  - Vercel: 7 USD

## System Health
- Dead daemons: 12 (down from 16 yesterday — 4 fixed)
- Public surfaces: 6 of 7 green (snowit.ba still NXDOMAIN)
- Email DLQ: 1 entry (from validation test)

Next: Phase 2 sprint planning (secondary tunnel + 12 dead daemons).

— John

Opt-out: Alem can pause digest via node ~/system/tools/mc.js config set digest.enabled false (not recommended — digest is designed to prevent morning blind spots).

Alert Routing by Source

BetterStack (External SaaS Monitors)

Monitor Name URL Check Interval Alert Channel Escalation
Drop Landing Page https://getdrop.no 3 min Slack #ops P0 if down > 10 min (revenue event)
alai.no Landing https://alai.no 3 min Slack #ops P0 if down > 5 min
lumiscare.alai.no https://lumiscare.alai.no 3 min Slack #ops P1 (demo, not production)
BookStack docs https://docs.alai.no 3 min Slack #ops P1 (internal wiki, not customer-facing)
Vaultwarden vault https://vault.alai.no 3 min Slack #ops P0 (email intake depends on it)
Documenso sign https://sign.alai.no 3 min Slack #ops P1 (signing, not immediate revenue)
snowit.ba https://snowit.ba 3 min Slack #ops P2 (currently NXDOMAIN, owner decision pending)

Alert message format from BetterStack:

[BetterStack] Monitor DOWN: <Monitor Name>
URL: <URL>
Status: <HTTP status code or DNS error>
Duration: <time since first failure>
Dashboard: https://betterstack.com/uptime

Cooldown: BetterStack has built-in "confirmation period" (30s) — waits 30s after first failure before firing alert (prevents transient network blip alerts).

ops-watchdog (Internal Daemon Monitors)

Service Check Type Interval Alert Channel Consecutive Failures Required
com.john.slack-bot PID check 2 min Email fallback (if dead, can't alert via Slack) 2
com.john.cloudflared PID + exit status 2 min Slack #ops + Email 2
com.john.ops-watchdog Self-health check 2 min Email (watchdog can't alert itself via Slack if dead) 2
com.john.email-agent PID + last-success file age 2 min Slack #ops 2
com.john.mc-dashboard PID + curl :3030 2 min Slack #ops 2
com.john.bookstack-sync PID 2 min Slack #ops 3 (less critical)
11 other critical daemons PID check 2 min Slack #ops 2

Public endpoint health checks (curl-based):

Endpoint Check Command Alert Channel Consecutive Failures
alai.no curl -sf https://alai.no | grep 'ALAI Holding' Slack #ops 2
lumiscare.alai.no curl -sf https://lumiscare.alai.no | grep 'LumisCare' Slack #ops 2
getdrop.no curl -sfL https://getdrop.no | grep 'Send penger' Slack #ops 2
docs.alai.no curl -sf https://docs.alai.no | grep 'BookStack' Slack #ops 2
vault.alai.no curl -sf https://vault.alai.no | grep 'Vaultwarden' Slack #ops 2
sign.alai.no curl -s -o /dev/null -w '%{http_code}' https://sign.alai.no | grep -E '^(200|301|302)' Slack #ops 2

Why 2 consecutive failures: Prevents false alerts from transient network hiccups. 2 failures = 4 min downtime before alert (2 min × 2 cycles).

Alert message format from ops-watchdog:

[SENTINEL ALERT] P<severity>: <Service Name> <Status>
Details: <exit code / curl error / PID missing>
Last check: 2026-04-19 10:24:15 CET
Config: ~/system/config/ops-watchdog.json
Runbook: ~/system/docs/runbooks/incident-response-playbook.md

HiveMind Event Bus (Agent-Generated Alerts)

Event Kind Subscriber Alert Channel Latency Example
kind=alert hivemind-alert-relay.js Slack #ops ≤ 10s Security scan fail, cost budget exceeded, agent loop detected
kind=intake hivemind-intake-mc-bridge.js MC auto-task + john-daily-digest ≤ 30s Email classified as support request, contact form submission
kind=briefing john-daily-digest Slack DM to Alem (08:00 CET) Daily Overnight summary, weekly report
kind=research (no subscriber yet) None n/a Agent research outcomes stored but not alerted
kind=skill_proposal john-daily-digest Slack DM to Alem (08:00 CET) Daily New skill added to library, cookbook entry

Alert message format from HiveMind:

[HM-ALERT] agent: <agent_name> | kind: <event_kind>
Message: <alert_message>
Timestamp: 2026-04-19T08:24:15Z
Evidence: <evidence_uri> (if available)
Action: <suggested_action> (if available)

Example:

[HM-ALERT] agent: securion-sentinel | kind: alert
Message: Public GitHub repo detected with potential ALAI internal code
Timestamp: 2026-04-19T08:24:15Z
Evidence: https://github.com/unknown-user/alai-leaked-repo
Action: Verify if repo is authorized OR issue DMCA takedown

TLS Cert Expiry Monitor (Scheduled Daily)

Domain Check Schedule Alert Thresholds Channel Escalation
alai.no Daily 07:00 CET 30d, 14d, 7d before expiry Slack #ops P0 at 7d (outage imminent)
lumiscare.alai.no Daily 07:00 CET 30d, 14d, 7d Slack #ops P1 (demo, not production)
getdrop.no Daily 07:00 CET 30d, 14d, 7d Slack #ops P0 (revenue app)
docs.alai.no Daily 07:00 CET 30d, 14d, 7d Slack #ops P1 (internal wiki)
vault.alai.no Daily 07:00 CET 30d, 14d, 7d Slack #ops P0 (email intake depends on it)
sign.alai.no Daily 07:00 CET 30d, 14d, 7d Slack #ops P1 (signing tool)
bilko-demo.basicconsulting.no Daily 07:00 CET 30d, 14d, 7d Slack #ops P2 (demo, not used — legacy cert)
snowit.ba Daily 07:00 CET 30d, 14d, 7d Slack #ops P2 (currently NXDOMAIN)
2 internal domains Daily 07:00 CET 30d, 14d, 7d Slack #ops P1

Alert message format:

[CERT-EXPIRY] P<severity>: <domain> expires in <days> days
Expiry date: <YYYY-MM-DD HH:MM:SS UTC>
Current cert issuer: <Let's Encrypt / Cloudflare / etc>
Action: Verify auto-renewal OR renew manually
Runbook: ~/system/docs/runbooks/incident-response-playbook.md#8-tls-cert-expiry

Why daily schedule: Cert renewal is not urgent (30d, 14d, 7d warnings). Checking every 2 min (like ops-watchdog) is wasteful. Daily check at 07:00 CET catches issues before business hours.

Alert Cooldowns & Rate Limiting

Goal: Prevent alert fatigue from flapping services or repeated failures.

Same-Alert Cooldown (15 min)

If same alert (same service + same failure type) fires within 15 min of previous alert → suppressed.

Example:

Exception: If service recovers and then fails again → new alert immediately (no cooldown on recovery → failure transition).

Repeated-Alert Escalation (5 alerts in 1 hour)

If same service fires > 5 alerts in 1 hour → escalate to P0 + tag Alem in Slack.

Example:

Email Fallback Rate Limit (10 emails per hour)

If Slack bot is dead and email fallback is active, limit emails to 10 per hour (prevents inbox flood during incident storm).

After 10 emails in 1 hour:

Buffered alerts replay to Slack #ops once bot is restored.

Which Daemons Send to Which Channel

Daemon Alert Channel Reason
com.john.ops-watchdog Slack #ops OR Email (if slack-bot dead) Core monitoring daemon — alerts about OTHER services
com.john.slack-bot Email only Can't alert itself via Slack (messenger is dead), must use email fallback
com.alai.john-daily-digest Slack DM to Alem Summary layer, not real-time alert
com.john.email-agent Slack #ops P1 if down (email intake stops)
com.john.cloudflared Slack #ops + Email P0 SPOF (26 hostnames die if tunnel down)
com.john.mc-dashboard Slack #ops P1 (internal dashboard, not customer-facing)
com.john.bookstack-sync Slack #ops P2 (wiki sync can lag 10 min without issue)
com.alai.cert-expiry-monitor Slack #ops P1 at 30d/14d, P0 at 7d
com.john.event-dispatcher Slack #ops P1 (HiveMind event bus — if dead, agent alerts stop flowing)
com.john.hook-daemon Slack #ops P0 (security enforcement — ZAKON NULA anti-hallucination gate)
7 other daemons Slack #ops P1 or P2 depending on criticality

Adding New Alert Routes

Step 1: Identify alert source (BetterStack, ops-watchdog, HiveMind, or new daemon).

Step 2: Determine severity (P0/P1/P2/P3) based on:

Step 3: Choose channel:

Step 4: Update routing config:

Step 5: Test alert delivery:

Cross-References

Evidence:

Alert routing maintained by: Skillforge (SENTINEL Task 7)
Last updated: 2026-04-19 (after SENTINEL sprint validation)
Next review: After Phase 2 sprint (secondary tunnel + 12 dead daemons fixed)

ALAI Hosting Operations

ALAI Hosting Operations Runbook

Owner: FlowForge (Kelsey Hightower) | Updated: 2026-07-28 | MC: #8491, #10218, #10221

1. Overview

This runbook covers operational procedures for ALAI's static site hosting on Cloudflare Pages. For architecture and migration plan, see the ALAI Static Hosting Blueprint (Infrastructure chapter).

In Scope:

Out of Scope:

Exception: vm-alai-support host-level storage (data disk, Docker data-root) is documented in Section 7, since no dedicated Azure VM runbook exists yet.

2. Rollback Procedure

When: Deploy caused production issue (5xx errors, broken UI, functionality regression)

Target: < 60 seconds from decision to live rollback

Step 1: Identify Last Known Good Deployment

# List recent deployments
npx wrangler pages deployment list --project-name=<project-name>

# Example output:
# ID: abc123def456
# Created: 2026-04-20 14:30:00
# Branch: main
# Status: active

Step 2: Execute Rollback

NOTE — wrangler 4.x breaking change: wrangler pages deployment rollback was removed in wrangler 4.x. The subcommand no longer exists and the /rollback CF API endpoint returns 405 for direct-upload deployments. Do NOT use it. Use the CF API re-deploy path below. (Reference: wrangler upstream release notes; verified in Proveo pilot on basicconsulting.no, MC #8494. See also ALAI Static Hosting Blueprint Section 3, "DR: Restore Site in < 60 Seconds".)

# CF API re-deploy (replaces deprecated wrangler rollback) — use ID from step 1
export CF_API_TOKEN="<your-cloudflare-api-token>"   # scope: Cloudflare Pages: Edit
export CF_ACCOUNT_ID="<your-cloudflare-account-id>"
export CF_PROJECT_NAME="<project-name>"

curl -s -X POST \
  "https://api.cloudflare.com/client/v4/accounts/${CF_ACCOUNT_ID}/pages/projects/${CF_PROJECT_NAME}/deployments/<deployment-id>/retry" \
  -H "Authorization: Bearer ${CF_API_TOKEN}" \
  -H "Content-Type: application/json" | python3 -c "import sys,json; r=json.load(sys.stdin); print('OK —', r['result']['id']) if r['success'] else print('ERROR:', r['errors'])"

# Example:
# curl -s -X POST ".../pages/projects/alai-no/deployments/abc123def456/retry" -H "Authorization: Bearer ${CF_API_TOKEN}" ...

Step 3: Verify

# Check HTTP status
curl -I https://<domain>

# Expected: HTTP/2 200
# If 5xx persists → escalate to L2 (Kelsey)

Step 4: Alert & Document

# Post to Slack
node ~/system/tools/slack.js send "#infra-alerts" \
  "ROLLBACK executed: <project-name> to deployment <deployment-id> at $(date -u +%Y-%m-%dT%H:%M:%SZ)"

# Create incident report (if > 5 min downtime)
node ~/system/tools/mc.js add "Incident: <domain> rollback" \
  --desc "Reason: [fill]. Rollback target: [deployment-id]. Downtime: [X min]" \
  --priority H --owner kelsey

3. SSL Certificate Auto-Renewal

Cloudflare Pages manages SSL certificates automatically via Cloudflare's CA. Certificates renew 30 days before expiry.

No manual action required.

Troubleshooting: SSL Cert Warning

If SENTINEL alerts "SSL cert expiry < 30 days":

# Step 1: Verify domain DNS points to Cloudflare
dig <domain> +short

# Expected: CNAME to <project-name>.pages.dev or Cloudflare IP range

# Step 2: Check Cloudflare dashboard
open "https://dash.cloudflare.com/pages"
# Navigate to: Project > Settings > Custom domains
# Verify: "SSL/TLS certificate" shows "Active"

# Step 3: If cert not renewing, trigger manual renewal
# (Cloudflare Pages does not expose manual renewal API — contact support)
node ~/system/tools/slack.js send "#infra-alerts" \
  "SSL cert not auto-renewing for <domain> — escalating to Cloudflare support"

4. Migration Workflow: New Site

Input: New static site needs hosting (markdown, React, Next.js static export, Astro)

Output: Site live on custom domain with SSL, SENTINEL monitoring enabled

Step 1: Validate Static Export

# For Next.js: verify static export enabled
grep 'output.*export' /path/to/site/next.config.js

# Expected: output: 'export'

# Build locally to verify
cd /path/to/site && npm run build

# Expected: Output directory exists (out/, dist/, .next/)

Step 2: Create Cloudflare Pages Project

# Option A: Dashboard (recommended for first-time)
open "https://dash.cloudflare.com/pages"
# Click: Create a project > Connect to Git > Select repo

# Option B: CLI
npx wrangler pages project create <project-name> --production-branch main

Step 3: Configure Build Settings

In Cloudflare dashboard: Project > Settings > Builds

Framework Build command Output directory
Static HTML (none) /
Next.js (static export) npm run build out
Astro npm run build dist

Save settings.

Step 4: Add GitHub Actions Workflow

Copy from template:

cp /Users/makinja/system/specs/templates/cf-pages-deploy.yml \
   /path/to/site/.github/workflows/deploy.yml

Commit and push to trigger first deploy.

Step 5: Add Custom Domain

# In Cloudflare dashboard: Project > Custom domains > Add custom domain
# Enter: <domain>

# If domain DNS is already on Cloudflare: CNAME record auto-created
# If domain DNS is external: Manual CNAME to <project-name>.pages.dev required

Verify SSL activates (usually < 5 min).

Step 6: Enable SENTINEL Monitoring

Add domain to /Users/makinja/system/tools/sentinel-uptime.sh:

# Open file
nano /Users/makinja/system/tools/sentinel-uptime.sh

# Add line to SITES array:
"https://<domain>"

# Save and test
bash /Users/makinja/system/tools/sentinel-uptime.sh

Verify Slack alert NOT sent (indicates site UP).

Step 7: Document

Update site inventory:

# Add line to ~/system/docs/infrastructure-inventory.md
echo "| <domain> | Cloudflare Pages | <project-name> | [GitHub repo URL] | ACTIVE |" \
  >> ~/system/docs/infrastructure-inventory.md

5. SENTINEL Uptime Integration

SENTINEL checks all ALAI sites every 5 minutes via cron.

Script: /Users/makinja/system/tools/sentinel-uptime.sh

Cron: */5 * * * * bash /Users/makinja/system/tools/sentinel-uptime.sh

Alert Channel: #infra-alerts (Slack)

Add New Site to SENTINEL

# Edit SITES array
nano /Users/makinja/system/tools/sentinel-uptime.sh

# Add:
"https://<domain>"

# Test manually
bash /Users/makinja/system/tools/sentinel-uptime.sh

# Expected: No output (site UP) or Slack alert (site DOWN)

Troubleshoot False Alerts

If SENTINEL reports DOWN but site is UP:

# Test from command line
curl -I --max-time 10 https://<domain>

# If returns 200: SENTINEL script has timeout issue (increase --max-time)
# If returns 5xx: Real issue — investigate Cloudflare Pages logs
# If returns 301/302: Update SENTINEL to accept redirects

6. Emergency DR: Serve from Azure VM

Scenario: Cloudflare Pages is down (e.g., Cloudflare incident) AND site is business-critical (e.g., alai.no during client demo).

Target: Site accessible within 120 seconds.

Step 1: Copy Build Output to VM

# From local machine:
cd /path/to/site
npm run build
scp -r ./out alai-admin@4.223.110.181:/var/www/<site-name>

Step 2: Serve via Caddy

# SSH to VM
ssh -i ~/.ssh/azure_alai alai-admin@4.223.110.181

# Start Caddy reverse proxy
sudo caddy reverse-proxy --from <domain> --to localhost:8080 &

# Start simple HTTP server
cd /var/www/<site-name> && python3 -m http.server 8080 &

Step 3: Update DNS (if needed)

# If Cloudflare DNS is also down, update DNS to point to Azure VM IP
# This requires registrar access — NOT recommended unless multi-hour Cloudflare outage

Step 4: Monitor & Rollback

# Verify site accessible
curl -I https://<domain>

# When Cloudflare recovers: DNS auto-reverts (CNAME to .pages.dev still exists)
# Kill Caddy process on VM
sudo killall caddy

7. Azure VM Data Disk Migration — vm-alai-support (MC #10218)

What: vm-alai-support's OS disk (30GB, Standard_B2als_v2) was filling up from Docker/containerd data (8 co-located services). CEO decision (Opcija 2, 2026-04-29) approved adding a dedicated managed data disk and moving the Docker data-root off the OS disk. Executed by FlowForge (Kelsey Hightower) 2026-04-30.

Services affected (all on vm-alai-support): bookstack, documenso, planka, baikal, vaultwarden, grafana, prometheus, caddy.

7.1 Disk Spec (tool-verified 2026-07-28 via az disk show)

Field Value
Disk name disk-alai-support-data
Resource group rg-alai-support (subscription 5b0b4d9b-e677-464e-abf0-5170cbce3b8e)
Region swedencentral
Size 128 GB
SKU StandardSSD_LRS (Standard SSD, tier "Standard")
Mount /mnt/data (/dev/sda1, ext4)
Attached to vm-alai-support
Created 2026-04-30T03:33:33Z
Docker daemon data-root /mnt/data/docker (/etc/docker/daemon.json)

⚠️ Discrepancy vs. task record: MC #10218's title and DoD evidence describe this as "Premium SSD P10 128GB / Premium_LRS." Live Azure state (checked 2026-07-28) shows the actual disk is StandardSSD_LRS (Standard SSD "E10" 128GB tier), not Premium. Cost/perf implications: Standard SSD is cheaper and lower-IOPS than Premium P10 — if a Premium disk was genuinely intended, this is an open gap; if Standard was chosen deliberately for cost, the task title/evidence text was simply inaccurate. Not re-litigated here — flagging so nobody cites "Premium P10" as fact downstream.

Mechanism note: the task title says "update docker-compose.yml bind paths," but /opt/alai/docker-compose.yml uses named Docker volumes (e.g. bookstack_config, caddy_data), not host bind paths. The actual migration mechanism was repointing the Docker daemon's data-root to /mnt/data/docker in /etc/docker/daemon.json — compose files were not edited for path changes.

7.2 Pre-requisite (verified before migration)

Azure Backup (RSV) first recovery point confirmed to exist before any migration step ran (MC #10210):

7.3 Migration Steps (as executed)

  1. Confirm RSV recovery point exists (#10210, above).
  2. az disk create --name disk-alai-support-data -g rg-alai-support ... (128GB).
  3. Attach disk to vm-alai-support (LUN0 per DoD evidence).
  4. SSH to VM, format new disk ext4, mount at /mnt/data.
  5. Stop Docker/containerd.
  6. rsync /var/lib/docker → /mnt/data/docker and /var/lib/containerd → /mnt/data/containerd.
  7. Set "data-root": "/mnt/data/docker" in /etc/docker/daemon.json.
  8. Start Docker/containerd; verify all 8 services (20 containers) come up healthy.
  9. Log phase completion: ~/system/specs/system-reform-status.log (DATA-DISK-MIGRATION | STARTED | MC:#10218).
  10. mc.js ready 10218 with DoD evidence; task marked done 2026-04-30 10:13:51.

Result (DoD evidence, MC #10218): 20 containers healthy post-migration. Old /var/lib/docker and /var/lib/containerd left in place pending a follow-up cleanup task (zero-FD check required first).

7.4 Old Directory Cleanup (7-day deletion window, MC #10228)

Live verification (2026-07-28): /var/lib/docker and /var/lib/containerd no longer exist on vm-alai-support — cleanup confirmed still in effect.

7.5 Current State (tool-verified 2026-07-28)

$ df -h / /mnt/data
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        29G   11G   19G  36% /
/dev/sda1       126G   19G  101G  16% /mnt/data

$ docker info | grep "Docker Root Dir"
Docker Root Dir: /mnt/data/docker

$ docker ps | wc -l
20

OS disk usage dropped from a pre-migration ~94% down to 36% (11G/29G) after the data-root move + old-dir cleanup. 20 containers running, matching the post-migration DoD count.

7.6 Rollback Procedure

No pre-written rollback script exists for this migration; the safety net during the 7-day window (before #10228 cleanup) was the untouched copy of /var/lib/docker and /var/lib/containerd on the OS disk plus the confirmed Azure Backup RSV recovery point (Section 7.2). After cleanup (2026-05-05), the OS-disk copies no longer exist — rollback today means either restoring from the Azure Backup RSV recovery point, or reversing the data-root pointer if /mnt/data itself is the failure:

A. Data disk corrupted / unmountable (VM boots, OS disk fine):

ssh -i ~/.ssh/azure_alai alai-admin@4.223.110.181
sudo systemctl stop docker
# point data-root back to local OS disk (temporary, degraded — no persisted volumes)
sudo sed -i 's#/mnt/data/docker#/var/lib/docker#' /etc/docker/daemon.json
sudo systemctl start docker
# Services will start with EMPTY volumes — this is a stop-gap to get containers running,
# not a data recovery step. Follow with Option B to recover actual data.

B. Full VM/data restore (disk lost, or data corruption needing point-in-time recovery):

# Restore from Azure Backup RSV (rsv-alai-support) to the recovery point closest
# to the desired time, or a later automatic RP if scheduled backups exist post-2026-04-29.
az backup restore restore-disks \
  --resource-group rg-alai-support --vault-name rsv-alai-support \
  --container-name "IaasVMContainer;iaasvmcontainerv2;rg-alai-support;vm-alai-support" \
  --item-name "VM;iaasvmcontainerv2;rg-alai-support;vm-alai-support" \
  --rp-name <recovery-point-name> --storage-account <staging-storage-account>
# Then attach recovered data disk to vm-alai-support, verify /mnt/data mount and
# docker data-root, restart services, verify all 8 services healthy (curl each domain).

Verify after any rollback:

curl -sI https://docs.alai.no | head -1   # BookStack
curl -sI https://vault.alai.no | head -1  # Vaultwarden
curl -sI https://sign.alai.no | head -1   # Documenso
curl -sI https://grafana.alai.no | head -1
curl -sI https://boards.alai.no | head -1 # Planka

8. Escalation

Issue L1 Action L2 Escalation L3 Escalation
Deploy failure Review build logs; check package.json/next.config.js Kelsey investigates Cloudflare Pages logs Contact Cloudflare support via dashboard
5xx errors (< 5 min) Execute rollback (Section 2) Kelsey reviews last commit for breaking change CEO notification + DR activation (Section 6)
SSL cert not renewing Verify DNS (Section 3) Kelsey triggers manual renewal or contacts CF support Switch to Let's Encrypt via Azure VM
SENTINEL false alerts Verify site UP via curl; adjust timeout Kelsey reviews SENTINEL script logic Disable SENTINEL for that site; use external monitor
DNS not resolving Verify Cloudflare DNS records; check registrar NS Kelsey checks registrar portal for NS change Contact registrar support

Key Contacts:

9. Maintenance Schedule

Task Frequency Owner How
Test rollback procedure Monthly Proveo (Angie Jones) Execute rollback on staging site; verify < 60s
Review SENTINEL alerts Weekly Kelsey Check Slack #infra-alerts for false positives
Update dependency versions Weekly Renovate bot Auto-merge minor/patch; manual review major
Backup DNS zone config Weekly Automated cron Exports to ~/system/backups/dns/
Verify SSL certs valid Daily SENTINEL Auto-alert if < 30 days to expiry

11. Change Log

Date Change Author
2026-04-20 Initial version — rollback, SSL, migration, SENTINEL Skillforge (MC #8491)
2026-07-28 Added Section 7 (Azure VM Data Disk Migration — vm-alai-support, MC #10218); flagged Premium_LRS vs. live StandardSSD_LRS discrepancy; renumbered Sections 8–11 Skillforge (MC #10221)
2026-07-28 Section 2 Step 2: replaced deprecated wrangler pages deployment rollback (removed in wrangler 4.x) with CF API re-deploy path, matching the blueprint fix already applied under MC #8494 FlowForge (MC #8494)

LightRAG Health Monitoring Runbook

LightRAG Health Monitoring Runbook

Domain note (2026-05-17): References to lightrag.basicconsulting.no and ollama.basicconsulting.no are legacy hostnames. Current live endpoints: lightrag.alai.no and ollama.alai.no.

Status: ACTIVE
Created: 2026-04-21
Owner: FlowForge (AgentForge)
Related: MC #8545, INFRA-CF-001

Purpose

Continuous health monitoring for LightRAG stack (Azure VM + Cloudflare) following the 2026-04-20 outage fix (CF Browser Integrity Check configuration).

This runbook covers:

Architecture Overview

LightRAG runs on Azure VM (20.240.61.67:9621) and is exposed via Cloudflare tunnel at https://lightrag.basicconsulting.no. The system depends on:

  1. Azure VM — Docker containers (lightrag + neo4j)
  2. Cloudflare tunnel — Routes traffic through Mac Studio relay
  3. Cloudflare Access — Authentication via service tokens
  4. Cloudflare BIC rule — Allows automation clients (Python UA)
  5. Ollama upstream — https://ollama.basicconsulting.no for LLM inference

See: Azure LightRAG Migration Runbook

Health Check Script

Location

~/system/tools/lightrag-health.sh

Manual Execution

bash ~/system/tools/lightrag-health.sh

Output

Exit Codes

Check Layers

Layer 1: Azure VM Health

Check What it tests Healthy criteria
direct_access Direct HTTP to VM IP:port HTTP 200, status=healthy
docker_containers Container status via SSH lightrag + neo4j running, healthy

Note: SSH access currently unavailable (publickey auth). Manual verification required via Azure Portal or after SSH key setup.

Layer 2: Cloudflare Network

Check What it tests Healthy criteria
cf_tunnel HTTPS via CF tunnel HTTP 200, latency < 2s
cf_bic_rule BIC rule configuration Rule enabled, covers both endpoints
python_ua Python client access HTTP 200 with Python UA

Critical: python_ua check verifies the CF-BIC-001 rule is active. If this fails with HTTP 403, automation clients (pi-orchestrator, lightrag-outbox-ingest.js) will break.

Layer 3: Application Health

Check What it tests Healthy criteria
health_endpoint /health endpoint status=healthy, pipeline_busy=false
query_endpoint /query with naive mode HTTP 200, valid response, < 30s

Note: First query after idle may take longer (cold start). If timeout, retry once.

Layer 4: Ollama Upstream

Check What it tests Healthy criteria
api_tags Ollama model availability qwen2.5-coder:32b + bge-m3 present

Critical: LightRAG requires these specific models. If missing, queries will fail.

Interpreting Results

Green (Exit 0) — Healthy

All critical checks passed. System operational.

Action: None required.

Yellow (Exit 1) — Warnings

Non-critical issues detected. System degraded but operational.

Common warnings:

Action: Review warning details. Monitor next check. Escalate if warnings persist 3+ checks.

Red (Exit 2) — Errors

Critical issues detected. System may be non-operational or partially failed.

Common errors:

Action:

  1. Review error details in JSON evidence
  2. Follow troubleshooting section below
  3. If unresolved after 30 min, consider rollback (see Azure LightRAG Migration Runbook)

Azure Monitor Alerts (MC #8803)

Status: ACTIVE / verified 2026-07-28
Scope: Azure VM vm-alai-lightrag in resource group rg-alai-lightrag (swedencentral)
Action group: lightrag-ops-oncall (alem@alai.no, john@alai.no)

Metric alerts

Alert Severity Metric Threshold Window / evaluation
lightrag-cpu-high 3 Percentage CPU Average > 85 15m / 5m
lightrag-memory-low 2 Available Memory Percentage Average < 15 15m / 5m
lightrag-disk-pressure 3 OS Disk IOPS Consumed Percentage Average > 80 15m / 5m
lightrag-vm-unavailable 1 VmAvailabilityMetric Minimum < 1 5m / 1m

VM state / administrative activity alerts

Alert Operation
lightrag-vm-power-state-change Microsoft.Compute/virtualMachines/deallocate/action
lightrag-vm-restart Microsoft.Compute/virtualMachines/restart/action
lightrag-vm-poweroff Microsoft.Compute/virtualMachines/powerOff/action

Verification commands

VM_ID="/subscriptions/5b0b4d9b-e677-464e-abf0-5170cbce3b8e/resourceGroups/rg-alai-lightrag/providers/Microsoft.Compute/virtualMachines/vm-alai-lightrag"

az monitor action-group list -g rg-alai-lightrag -o table
az monitor metrics alert list -g rg-alai-lightrag -o table
az monitor activity-log alert list -g rg-alai-lightrag -o table
az monitor metrics list --resource "$VM_ID" \
  --metric "Percentage CPU,Available Memory Percentage,OS Disk IOPS Consumed Percentage,VmAvailabilityMetric" \
  --interval PT1M --aggregation Average -o table
az vm run-command invoke -g rg-alai-lightrag -n vm-alai-lightrag \
  --command-id RunShellScript --scripts 'df -h / /var/lib/docker || df -h /; free -m; uptime'

Disk caveat: Azure platform metrics expose OS disk IOPS/bandwidth pressure, not filesystem free-space percentage. Filesystem usage must be checked via guest telemetry (df) or a future AMA/Log Analytics/custom-metric extension if disk-full alerting is required.

Automated Monitoring Setup

LaunchAgent Installation (DRAFT — Pending Alem Approval)

Draft file: ~/system/evidence/lightrag-monitor-launchagent-draft.plist

Schedule: Daily at 9:00 AM (frequent for 4-week monitoring period)

Installation steps (when approved):

# 1. Copy draft to LaunchAgents
cp ~/system/evidence/lightrag-monitor-launchagent-draft.plist \
   ~/Library/LaunchAgents/com.john.lightrag-monitor.plist

# 2. Load the agent
launchctl load ~/Library/LaunchAgents/com.john.lightrag-monitor.plist

# 3. Start immediately (optional)
launchctl start com.john.lightrag-monitor

Manual trigger:

launchctl kickstart -k gui/$(id -u)/com.john.lightrag-monitor

Logs:

Slack Alerts (To Be Implemented)

When LaunchAgent detects exit code 2 (errors), send alert to #alerts channel:

node ~/system/tools/slack.js send alerts "🚨 LightRAG health check FAILED at $(date). Check ~/system/evidence/lightrag-health-*.json"

This requires wrapping the health check script in a post-execution hook (see plist comments).

Health History Database

Location: ~/system/databases/lightrag-health.db

Schema: ~/system/tools/lightrag-health-db-init.sql

Tables

Views

Query Examples

Last 10 checks:

sqlite3 ~/system/databases/lightrag-health.db \
  "SELECT timestamp, overall_status, errors, warnings FROM health_checks ORDER BY created_at DESC LIMIT 10;"

Trend over last 7 days:

sqlite3 ~/system/databases/lightrag-health.db \
  "SELECT * FROM health_checks_trend WHERE check_date >= date('now', '-7 days');"

All errors in last 24 hours:

sqlite3 ~/system/databases/lightrag-health.db \
  "SELECT hc.timestamp, hcd.layer, hcd.check_name, hcd.message FROM health_checks hc
   JOIN health_check_details hcd ON hc.id = hcd.health_check_id
   WHERE hcd.status = 'error' AND hc.created_at >= datetime('now', '-24 hours');"

Note: Database logging will be implemented in next iteration of the health script.

Troubleshooting

Issue: Query endpoint timeout (HTTP 000, 35s)

Possible causes:

  1. First query after idle (cold start)
  2. Ollama FORGE overloaded
  3. Network path issue (Mac Studio → CF → Azure → CF → Mac Studio)

Diagnosis:

# Test Ollama upstream directly
curl -s https://ollama.basicconsulting.no/api/tags \
  -H "CF-Access-Client-Id: $(grep CF_ACCESS_CLIENT_ID ~/Library/LaunchAgents/com.john.pi-orchestrator.plist | sed 's/.*<string>\(.*\)<\/string>/\1/')" \
  -H "CF-Access-Client-Secret: $(grep CF_ACCESS_CLIENT_SECRET ~/Library/LaunchAgents/com.john.pi-orchestrator.plist | sed 's/.*<string>\(.*\)<\/string>/\1/')" | jq '.models | length'

# Check if FORGE is responding
curl http://10.0.0.2:11434/api/ps

# Test query directly with extended timeout
curl -s --max-time 60 \
  -H "CF-Access-Client-Id: ..." \
  -H "CF-Access-Client-Secret: ..." \
  -H "Content-Type: application/json" \
  -X POST \
  -d '{"query":"test","mode":"naive","only_need_context":false}' \
  https://lightrag.basicconsulting.no/query

Fix:

Issue: Python UA blocked (HTTP 403)

Root cause: CF Browser Integrity Check rule disabled or misconfigured.

Diagnosis:

# Test with Python UA
curl -s -w "\nHTTP: %{http_code}\n" \
  -A "Python/3.11 urllib/1.26" \
  -H "CF-Access-Client-Id: ..." \
  -H "CF-Access-Client-Secret: ..." \
  https://lightrag.basicconsulting.no/health

Fix:

  1. Verify CF Configuration Rule (Ruleset 4fc2c122d04d4791a5d17409b097c510, Rule c5990f19f655441180ae886f4512de40)
  2. Ensure rule is enabled and expression includes lightrag.basicconsulting.no
  3. See: ~/system/rules/cf-proxied-api-bic-whitelist.md

Critical: This is a repeat of the 2026-04-20 outage. If rule is disabled, all automation breaks.

Issue: Ollama models missing

Symptoms: api_tags check fails or warns about missing models.

Required models:

Fix:

# SSH to FORGE (10.0.0.2)
ssh admin@10.0.0.2

# Pull missing models
ollama pull qwen2.5-coder:32b-instruct-q8_0
ollama pull bge-m3:latest

# Verify
ollama list | grep -E "(qwen2.5-coder:32b-instruct-q8_0|bge-m3:latest)"

Issue: Direct VM access failed

Symptoms: direct_access check returns HTTP error or timeout.

Diagnosis:

# Test direct HTTP
curl -s --connect-timeout 5 http://20.240.61.67:9621/health

# Check NSG rules (Mac Studio IP may have changed)
az network nsg rule show \
  -g rg-alai-lightrag \
  --nsg-name vm-alai-lightragNSG \
  -n allow-lightrag-macstudio \
  --query "sourceAddressPrefix"

# Compare to current ISP IP
curl -s https://ifconfig.co

Fix: If Mac Studio ISP IP rotated, update NSG rule:

NEW_IP=$(curl -s https://ifconfig.co)
az network nsg rule update \
  -g rg-alai-lightrag \
  --nsg-name vm-alai-lightragNSG \
  -n allow-lightrag-macstudio \
  --source-address-prefixes "${NEW_IP}/32"

Note: Azure resources (rg-alai-lightrag) are not currently visible via az CLI. This may indicate different subscription or access issue. Direct HTTP access confirms VM is operational.

Rollback Procedure

If LightRAG stack becomes unstable (exit code 2 persisting > 30 min, or CEO directive):

Follow: Azure LightRAG Migration Runbook → Section "Rollback Procedure"

Summary:

  1. Revert consumer URLs from https://lightrag.basicconsulting.no to http://localhost:9621
  2. Restart local Docker LightRAG
  3. Verify local service
  4. Optionally deprovision Azure VM

Expected rollback time: 5-15 minutes
Data loss risk: ZERO (local volumes preserved)

Maintenance

Weekly Tasks (First 4 Weeks)

After 4 Weeks

If system stable (no exit code 2 in 4 weeks):

Evidence Files

All health checks generate timestamped evidence:

Location: ~/system/evidence/lightrag-health-YYYYMMDD-HHMMSS.*

Retention: Keep last 30 days, archive older to Azure Blob Storage.

Example archive command (to be automated):

find ~/system/evidence -name "lightrag-health-*.json" -mtime +30 \
  | xargs tar -czf ~/system/evidence/archive-$(date +%Y%m).tar.gz

# Upload to Azure Blob
az storage blob upload \
  --account-name plockfrontstaging \
  --container-name evidence \
  --name lightrag-health-archive-$(date +%Y%m).tar.gz \
  --file ~/system/evidence/archive-$(date +%Y%m).tar.gz

Changelog

2026-04-21 — Initial version (baseline setup + first run)

Document Owner: FlowForge
Last Updated: 2026-04-21
Approved By: Pending Alem approval for LaunchAgent installation

GCP Auth Runbook — alai-cli-deployer SA (MC #9522)

GCP Auth Runbook (post-MC #9522)

Status

Active as of 2026-04-26. SA key created, activated, verified.

Primary auth

Daily use

No login needed. gcloud commands authenticate via SA key automatically. The SA is set as default account: gcloud config set account alai-cli-deployer@...

Verification at any time:

gcloud auth list   # SA shows ACTIVE (*)
gcloud run services list --project tribal-sign-487920-k0 --region europe-north1

Key rotation (every 90 days — next due 2026-07-26)

# 1. Get existing key IDs
gcloud iam service-accounts keys list \
  --iam-account=alai-cli-deployer@tribal-sign-487920-k0.iam.gserviceaccount.com \
  --project=tribal-sign-487920-k0

# 2. Create new key (requires org policy exception — see below)
gcloud iam service-accounts keys create ~/.gcloud/alai-cli-deployer-new.json \
  --iam-account=alai-cli-deployer@tribal-sign-487920-k0.iam.gserviceaccount.com \
  --project=tribal-sign-487920-k0
chmod 0600 ~/.gcloud/alai-cli-deployer-new.json

# 3. Activate new key
gcloud auth activate-service-account \
  alai-cli-deployer@tribal-sign-487920-k0.iam.gserviceaccount.com \
  --key-file=/Users/makinja/.gcloud/alai-cli-deployer-new.json

# 4. Test it works
gcloud run services list --project tribal-sign-487920-k0 --region europe-north1

# 5. Delete old key (use KEY_ID from step 1)
gcloud iam service-accounts keys delete OLD_KEY_ID \
  --iam-account=alai-cli-deployer@tribal-sign-487920-k0.iam.gserviceaccount.com \
  --project=tribal-sign-487920-k0

# 6. Swap file and update Bitwarden
mv ~/.gcloud/alai-cli-deployer-new.json ~/.gcloud/alai-cli-deployer.json
# Update Bitwarden item with new key content (bw edit item <ID>)

Org policy note (IMPORTANT for key rotation)

The org policy constraints/iam.disableServiceAccountKeyCreation is enforced org-wide. To create a new key during rotation, temporarily allow at project level:

# 1. Allow (run as dev@alai.no)
gcloud config set account dev@alai.no
cat > /tmp/policy-allow.yaml << 'YAML'
name: projects/762788903040/policies/iam.disableServiceAccountKeyCreation
spec:
  rules:
  - enforce: false
YAML
gcloud org-policies set-policy /tmp/policy-allow.yaml

# 2. Wait ~30-90s for propagation, then create key (see rotation steps above)

# 3. Restore restriction after key created
gcloud org-policies delete constraints/iam.disableServiceAccountKeyCreation \
  --project=tribal-sign-487920-k0

# 4. Switch back to SA account
gcloud config set account alai-cli-deployer@tribal-sign-487920-k0.iam.gserviceaccount.com

Recovery (if key file lost)

  1. CEO Alem: gcloud auth login with dev@alai.no (one-time interactive)
  2. Retrieve key from Bitwarden: bw get item "GCP Service Account Key — alai-cli-deployer" --session $(cat /tmp/bw-session) | jq -r .notes > ~/.gcloud/alai-cli-deployer.json && chmod 0600 ~/.gcloud/alai-cli-deployer.json
  3. Activate: gcloud auth activate-service-account alai-cli-deployer@tribal-sign-487920-k0.iam.gserviceaccount.com --key-file=/Users/makinja/.gcloud/alai-cli-deployer.json
  4. Set default: gcloud config set account alai-cli-deployer@tribal-sign-487920-k0.iam.gserviceaccount.com
  5. Verify: gcloud run services list --project tribal-sign-487920-k0 --region europe-north1

Recovery (if Bitwarden unavailable — last resort)

If key file AND Bitwarden lost, follow key rotation procedure:

  1. CEO Alem runs gcloud auth login (one-time interactive)
  2. Apply org policy override (see above)
  3. Create new key
  4. Activate, store in Bitwarden, restore policy

Future work

Security notes

Azure Auth Runbook — alai-cli-deployer SP (MC #9524)

Azure Auth Runbook (post-MC #9524)

Status

Active as of 2026-04-26. SP created, authenticated, verified.

Primary auth

Daily use

No login needed. az commands authenticate via SP token automatically. Token TTL is 1 hour but renewed silently by az CLI — no interactive prompt.

Verification at any time:

az account show --query "{user:user.name,type:user.type}"
# Expected: {"user": "f2a3b94b-46a5-4a5c-ae34-a222a35bf5b9", "type": "servicePrincipal"}
az vm list --query "[].name"
# Expected: ["repair-vm-alai_", "vm-alai-lightrag", "vm-alai-support", "vm-drop-prod"]

Covered resources

VM Resource Group Purpose
vm-alai-support rg-alai-support BookStack, Vaultwarden, Documenso, Grafana, Planka
vm-drop-prod RG-DROP-PROD Drop production
vm-alai-lightrag rg-alai-lightrag LightRAG knowledge graph
repair-vm-alai_ repair-vm-alai-support-... Ephemeral repair VM

SSH still uses key-based auth: ssh -i ~/.ssh/azure_alai alai-admin@4.223.110.181

SP secret rotation (every 90 days — next due 2026-07-26)

# 1. Retrieve current SP secret from Bitwarden (for reference)
BW_SESSION=$(cat /tmp/bw-session)
bw get item "Azure Service Principal — alai-cli-deployer" --session "$BW_SESSION" | jq -r .notes

# 2. Create new secret (requires user account with AD rights — alem@alai.no)
az login  # one-time interactive as alem@alai.no
az ad sp credential reset \
  --id "f2a3b94b-46a5-4a5c-ae34-a222a35bf5b9" \
  --years 2 \
  2>&1
# → returns new password

# 3. Test new secret
az login \
  --service-principal \
  -u "f2a3b94b-46a5-4a5c-ae34-a222a35bf5b9" \
  -p "<NEW_PASSWORD>" \
  --tenant "3454a03f-20b4-4bda-a116-2293c459aecd"
az vm list --query "[].name"

# 4. Update Bitwarden item with new secret
# bw edit item 7865a3a3-c4af-4aef-ac68-8dce370b5010 --session "$BW_SESSION" (update notes field)

# 5. Update rotation_due date in this file and in infra_service_account_auth_pattern.md

Recovery (if SP secret unknown)

  1. Alem: az login with alem@alai.no (one-time interactive)
  2. Reset SP: az ad sp credential reset --id f2a3b94b-46a5-4a5c-ae34-a222a35bf5b9 --years 2
  3. Re-login as SP with new secret
  4. Update Bitwarden item
  5. Verify: az vm list --query "[].name"

Recovery (if Bitwarden unavailable — last resort)

  1. Alem: az login (one-time interactive, alem@alai.no)
  2. az ad sp credential reset --id f2a3b94b-46a5-4a5c-ae34-a222a35bf5b9 --years 2 → new secret
  3. az login --service-principal -u f2a3b94b-46a5-4a5c-ae34-a222a35bf5b9 -p <new> --tenant 3454a03f-20b4-4bda-a116-2293c459aecd
  4. Store new secret in Bitwarden when available
  5. Update this runbook

Activate from scratch (fresh machine)

# 1. Retrieve secret from Bitwarden
BW_SESSION=$(bw unlock --raw)
SECRET=$(bw get item "Azure Service Principal — alai-cli-deployer" --session "$BW_SESSION" | \
  python3 -c "import sys,json; n=json.load(sys.stdin)['notes']; [print(l.split(': ',1)[1]) for l in n.split('\n') if l.startswith('password')]")

# 2. Login
az login \
  --service-principal \
  -u "f2a3b94b-46a5-4a5c-ae34-a222a35bf5b9" \
  -p "$SECRET" \
  --tenant "3454a03f-20b4-4bda-a116-2293c459aecd"

# 3. Verify
az account show --query "{user:user.name,type:user.type}"

Security notes

AWS Auth Runbook — alai-cli-deployer IAM key (MC #9523)

AWS Auth Runbook (post-MC #9523)

Status

Active as of 2026-04-27. IAM access key created, activated, verified.

Primary auth

Shell activation

AWS_PROFILE=alai-cli-deployer is exported in ~/.zshrc (added MC #9523, 2026-04-27). No interactive login needed. All aws commands use this profile by default.

Override for a single command: AWS_PROFILE=alai-cli-deployer aws

Daily use

No login needed. All aws CLI commands authenticate via the access key in ~/.aws/credentials.

Verification at any time: aws sts get-caller-identity Expected: UserId AIDAUXDEHCNUHSS72WSYC, Arn arn:aws:iam::324480209768:user/alai-cli-deployer

aws apprunner list-services --region eu-west-1 aws ecr describe-repositories --region eu-west-1

IAM Policies (as of MC #9523, 2026-04-27)

Policy | Rationale AWSAppRunnerFullAccess | Drop deploy - create/update/start App Runner services AmazonEC2ContainerRegistryFullAccess | Push/pull Docker images to ECR (Drop API + Web) SecretsManagerReadWrite | Read/write Drop secrets (DB, API keys) AmazonS3FullAccess | Build artifacts, CodeBuild source/output buckets CloudWatchLogsFullAccess | App Runner + CodeBuild runtime logs AWSCodeBuildAdminAccess | MC #9540 Drop CodeBuild (future)

Key rotation (every 90 days - next due 2026-07-26)

  1. Create new access key: aws iam create-access-key --user-name alai-cli-deployer > /tmp/new-key.json

  2. Update credentials file (use Python, do NOT print secret to terminal): python3 -c " import os, json, configparser new = json.load(open('/tmp/new-key.json'))['AccessKey'] cfg_path = os.path.expanduser('~/.aws/credentials') config = configparser.ConfigParser() config.read(cfg_path) config['alai-cli-deployer']['aws_access_key_id'] = new['AccessKeyId'] config['alai-cli-deployer']['aws_secret_access_key'] = new['SecretAccessKey'] with open(cfg_path, 'w') as f: config.write(f) os.chmod(cfg_path, 0o600) print('Updated:', new['AccessKeyId']) "

  3. Verify: AWS_PROFILE=alai-cli-deployer aws sts get-caller-identity

  4. Delete old key: aws iam delete-access-key --user-name alai-cli-deployer --access-key-id OLD_KEY_ID

  5. Update Bitwarden item 0605acce-fb80-4a36-ac11-3b55ffe66a3e with new key values

  6. Shred temp file: shred -u /tmp/new-key.json

Recovery (if credentials file lost)

  1. Retrieve key from Bitwarden: SESSION=0L9KMqYMX1/HfMdDBLJ3MsNZwATGz5Bv++fCFat2uT1RPCrvy1mCrcsNiL0uGxeiyTIJXKWkWV28W0vjZEjq4A== BW_SESSION= bw --nointeraction get item 0605acce-fb80-4a36-ac11-3b55ffe66a3e | jq -r '.login.username, .login.password'

  2. Re-create ~/.aws/credentials profile with recovered values (mode 0600)

  3. Verify: AWS_PROFILE=alai-cli-deployer aws sts get-caller-identity

Recovery (if Bitwarden unavailable - last resort)

  1. Authenticate as a user with IAM admin access
  2. Create new access key: aws iam create-access-key --user-name alai-cli-deployer
  3. Update credentials file + Bitwarden
  4. Delete old key after verification

Security notes

Services accessible with this profile

CF IP Access Rules — ALAI LAN Bypass

CF IP Access Rules — ALAI LAN Bypass

Zone: alai.no Zone ID: 3dc40d9c37fee79c4281f7e86870c0b5 Last updated: 2026-04-28 MC reference: #9956

Active rules

Rule ID IP Mode Created Notes
94994e3badcd4349815190038940bf19 92.221.168.61/32 whitelist 2026-04-28 ALAI LAN egress (Klofta) — Mac Studio/ANVIL + Mac Air + peers

Why this exists

ALAI internal automation (Python klijenti, curl skripte, CI agenti) konektujući se na *.alai.no servise iz ALAI LAN egress IP-a hit-ovali su CF WAF/bot detection (error 1010), uzrokujući 46h LightRAG outage 2026-04-20 i konstantne automation failures. IP Access Rule sa mode=whitelist suprimira WAF/bot blocks za saobraćaj iz ovog IP-a.

Kako radi (CF layer order)

  1. Request stiže na CF edge
  2. IP Access Rules se evaluiraju — ako IP match-uje whitelist, WAF/bot je bypassed
  3. CF Access (Zero Trust) se evaluira — auth redirect i dalje važi bez obzira na IP whitelist
  4. Origin reached

So: whitelist suprimira WAF/bot, ali NE preskače CF Access autentikaciju. Dva nezavisna sloja.

Authoritative IP source — NE koristi curl ifconfig.me sam

Per zakon-network-egress-verification.md:

3-source verifikacija obavezna prije bilo kakvog whitelist task-a.

Verifikacija

Iz whitelistovanog IP-a

curl -sI https://lightrag.alai.no/
# Expected: HTTP 200 (ili 302 redirect na CF Access — oboje OK, no 1010)

Provjera da rule postoji u CF

TOKEN=$(bw get item "Cloudflare Global API Key" --session $(cat /tmp/bw-session))
curl -s "https://api.cloudflare.com/client/v4/zones/3dc40d9c37fee79c4281f7e86870c0b5/firewall/access_rules/rules?configuration.value=92.221.168.61" \
  -H "X-Auth-Email: ..." -H "X-Auth-Key: $TOKEN" | jq

Lista svih IP Access Rules

curl -s "https://api.cloudflare.com/client/v4/zones/3dc40d9c37fee79c4281f7e86870c0b5/firewall/access_rules/rules" \
  -H "X-Auth-Email: ..." -H "X-Auth-Key: $TOKEN" | jq '.result[] | {id, mode, configuration, notes}'

Dodavanje novog IP-a u whitelist

  1. 3-source verifikacija — Mehanik Phase N gate to enforces:
    • VPN check: ifconfig | grep -c "^utun"
    • Source 1: curl -s https://api.ipify.org
    • Source 2: dig +short myip.opendns.com @resolver1.opendns.com
    • Source 3: tailscale status | grep "direct"
  2. POST CF API: 
    curl -X POST \
  "https://api.cloudflare.com/client/v4/zones/3dc40d9c37fee79c4281f7e86870c0b5/firewall/access_rules/rules" \
  -H "X-Auth-Email: ..." -H "X-Auth-Key: $TOKEN" \
  -d '{"mode": "whitelist", "configuration": {"target": "ip", "value": "<NEW_IP>"}, "notes": "..."}'
  3. Validation: curl iz whitelistovanog IP-a, expect 200
  4. Update ovog dokumenta i DEPLOY-MAP.md sa novim Rule ID + IP

Out of scope za whitelist

CF IP Access Rules — ALAI LAN Bypass

CF IP Access Rules — ALAI LAN Bypass

Zone: alai.no Zone ID: 3dc40d9c37fee79c4281f7e86870c0b5 Account ID: d0ac2afb6bb5b298723b85a114151a04 Last updated: 2026-04-28 MC references: #9956 (WAF whitelist), #9546 (CF Access bypass)

Two CF layers, two rule types — both require ALAI LAN egress (92.221.168.61) in their respective allowlists.

Layer 1 — IP Access Rules (WAF / bot / rate-limit bypass)

API: /zones/{zone_id}/firewall/access_rules/rules

Rule ID IP Mode Created Notes
94994e3badcd4349815190038940bf19 92.221.168.61/32 whitelist 2026-04-28 ALAI LAN egress (Klofta) — Mac Studio/ANVIL + Mac Air + peers

Effect: Suppresses CF WAF/bot detection (error 1010), rate-limit, security level checks. Auth gates (CF Access) still apply.

Layer 2 — CF Access (Zero Trust auth bypass) policies

API: /accounts/{account_id}/access/apps/{app_id}/policies/{policy_id}

App Name App ID Domain Policy ID Decision IPs in include
All ALAI Services cd7cf0f0-ab37-4b06-8d51-9f042fd7a4f6 *.alai.no cecc0b27-192e-4d09-be80-27d792945a60 bypass 46.46.253.33, 20.240.61.67, 46.46.251.40, 46.46.247.60, 92.221.168.61
All Studio Services f4d85fab-1c4b-4a48-97a6-ea982e7444e2 *.basicconsulting.no b72086cd-2995-403e-872f-b2c29d3aac39 bypass same 5 IPs
lightrag.alai.no c62b46b1-43f4-4967-9b99-cabfefb6b99b lightrag.alai.no 913922cd-c637-4999-b0e9-ef25f9e35fae bypass 46.46.253.33, 20.240.61.67, 46.46.251.40, 92.221.168.61
ollama.alai.no bdc17e6a-94c2-42d9-b5ce-37c1c37ac016 ollama.alai.no 162b2533-bd29-497e-9ada-db8684da869d bypass 46.46.253.33, 20.240.61.67, 46.46.251.40, 92.221.168.61

Effect: Skips Zero Trust auth (302 redirect to cloudflareaccess.com). Direct backend response.

Auth: Cloudflare Global API Key (john@basicconsulting.no) via Bitwarden — required for /access/* endpoints (regular cf-api-token insufficient scope).

Why this exists

ALAI internal automation (Python klijenti, curl skripte, CI agenti) konektujući se na *.alai.no servise iz ALAI LAN egress IP-a hit-ovali su CF WAF/bot detection (error 1010), uzrokujući 46h LightRAG outage 2026-04-20 i konstantne automation failures. IP Access Rule sa mode=whitelist suprimira WAF/bot blocks za saobraćaj iz ovog IP-a.

Kako radi (CF layer order)

  1. Request stiže na CF edge
  2. IP Access Rules se evaluiraju — ako IP match-uje whitelist, WAF/bot je bypassed
  3. CF Access (Zero Trust) se evaluira — auth redirect i dalje važi bez obzira na IP whitelist
  4. Origin reached

So: whitelist suprimira WAF/bot, ali NE preskače CF Access autentikaciju. Dva nezavisna sloja.

Authoritative IP source — NE koristi curl ifconfig.me sam

Per zakon-network-egress-verification.md:

3-source verifikacija obavezna prije bilo kakvog whitelist task-a.

Verifikacija

Iz whitelistovanog IP-a

curl -sI https://lightrag.alai.no/
# Expected: HTTP 200 (ili 302 redirect na CF Access — oboje OK, no 1010)

Provjera da rule postoji u CF

TOKEN=$(bw get item "Cloudflare Global API Key" --session $(cat /tmp/bw-session))
curl -s "https://api.cloudflare.com/client/v4/zones/3dc40d9c37fee79c4281f7e86870c0b5/firewall/access_rules/rules?configuration.value=92.221.168.61" \
  -H "X-Auth-Email: ..." -H "X-Auth-Key: $TOKEN" | jq

Lista svih IP Access Rules

curl -s "https://api.cloudflare.com/client/v4/zones/3dc40d9c37fee79c4281f7e86870c0b5/firewall/access_rules/rules" \
  -H "X-Auth-Email: ..." -H "X-Auth-Key: $TOKEN" | jq '.result[] | {id, mode, configuration, notes}'

Dodavanje novog IP-a u whitelist

  1. 3-source verifikacija — Mehanik Phase N gate to enforces:
    • VPN check: ifconfig | grep -c "^utun"
    • Source 1: curl -s https://api.ipify.org
    • Source 2: dig +short myip.opendns.com @resolver1.opendns.com
    • Source 3: tailscale status | grep "direct"
  2. POST CF API: 
    curl -X POST \
  "https://api.cloudflare.com/client/v4/zones/3dc40d9c37fee79c4281f7e86870c0b5/firewall/access_rules/rules" \
  -H "X-Auth-Email: ..." -H "X-Auth-Key: $TOKEN" \
  -d '{"mode": "whitelist", "configuration": {"target": "ip", "value": "<NEW_IP>"}, "notes": "..."}'
  3. Validation: curl iz whitelistovanog IP-a, expect 200
  4. Update ovog dokumenta i DEPLOY-MAP.md sa novim Rule ID + IP

Out of scope za whitelist

archive.alai.no — Paperless-ngx Setup & Operations

archive.alai.no — Paperless-ngx Setup & Operations

URL: https://archive.alai.no Backend: Paperless-ngx (image ghcr.io/paperless-ngx/paperless-ngx:latest) Host: Azure VM 4.223.110.181 (alai-admin) Container: alai-paperless-1 (with redis, gotenberg, tika sidecars) MC reference: #9546, #9982 (DR backup TODO)

Document management system za sve ALAI-srodne legalne, ugovorne, partnerske, istraživačke i finansijske dokumente. OCR, full-text search, taxonomy.

Access requirements

CF stack (oba sloja) traže 92.221.168.61/32 (ALAI LAN egress) u bypass listama. Vidi CF IP Access Rules — ALAI LAN Bypass.

Iz Mac Studio sa aktivnim VPN-om: bind interface 192.168.68.65 (Deco LAN) zaobilazi VPN routing:

curl --interface 192.168.68.65 https://archive.alai.no/...

Mac Air i ostali bez VPN-a: direktno radi.

API authentication

Paperless koristi DRF Token auth.

Token za admin user (root@localhost) sačuvan lokalno na Mac Studio:

~/.config/alai/paperless-token.env  (mode 600)

PAPERLESS_TOKEN=c9ec30192db3c95802349335edea4bca864a937a
PAPERLESS_BASE=https://archive.alai.no
PAPERLESS_BIND_INTERFACE=192.168.68.65

Svi API zahtjevi:

Authorization: Token c9ec30192db3c95802349335edea4bca864a937a

Regenerate token (ako compromised — Django shell preko docker exec):

ssh -b 192.168.68.65 -i ~/.ssh/azure_alai alai-admin@4.223.110.181 \
  'docker exec alai-paperless-1 python manage.py shell -c "
from rest_framework.authtoken.models import Token
from django.contrib.auth import get_user_model
u = get_user_model().objects.get(username=\"admin\")
Token.objects.filter(user=u).delete()
print(Token.objects.create(user=u).key)
"'

Schema (taxonomy)

Setup-ovan 2026-04-28 preko /tmp/paperless-setup.sh. ID-evi mogu varirati po instanci — koristi name__iexact za lookup.

Document Types (14 base, currently 25 active)

Contract, LOI, NDA, Registration, Insurance Policy, Research Paper, Invoice, Receipt, Email Archive, Identity Document, Tax Document, Financial Statement, Meeting Notes, Pitch Deck — plus historical types from prior usage. Numbers grow naturally; verify current via API.

Tags (23 base, currently 39 active, color-coded)

Cross-cutting (cilj): legal, research, kuran-19, partnership, regulator, contract, nda, loi, invoice, registration, urgent, signed, pending-signature

Company tags: ALAI, Drop, Bilko, Tok, Lobby, LumisCare, Plock, ALAI-Tech-DOO, BasicConsulting, client

Storage Paths (21)

Folder hijerarhija po kompaniji + funkciji:

/ALAI/legal/{created_year}/{title}
/ALAI/research/kuran-19/{title}
/ALAI/research/general/{created_year}/{title}
/ALAI/partnerships/sintef/{title}
/ALAI/partnerships/intesa/{title}
/ALAI/partnerships/pbz/{title}
/ALAI/regulators/finanstilsynet/{created_year}/{title}
/ALAI/regulators/skatteetaten/{created_year}/{title}
/ALAI/regulators/bronnoysund/{created_year}/{title}
/ALAI/contacts/{title}
/Drop/legal/{created_year}/{title}
/Drop/contracts/{title}
/Bilko/legal/{created_year}/{title}
/Bilko/contracts/{title}
/Tok/legal/{created_year}/{title}
/Lobby/legal/{created_year}/{title}
/LumisCare/legal/{created_year}/{title}
/Plock/legal/{created_year}/{title}
/ALAI-Tech-DOO/legal/{created_year}/{title}
/BasicConsulting/{created_year}/{title}
/clients/Entur/{created_year}/{title}

Initial Correspondents (11 seeded, currently 25 active, auto-expand)

SINTEF, Finanstilsynet, Skatteetaten, Brønnøysundregistrene, PBZ Zagreb, Intesa Sanpaolo, Anthropic, Cloudflare, Tryg, Fiken AS, Entur AS — auto-create on classify match.

Upload workflow

Manual single file

source ~/.config/alai/paperless-token.env
curl -s --interface "$PAPERLESS_BIND_INTERFACE" \
  -H "Authorization: Token $PAPERLESS_TOKEN" \
  -F "title=My Document" \
  -F "storage_path=1" \
  -F "tags=30" -F "tags=17" \
  -F "document=@/path/to/file.pdf" \
  -X POST "$PAPERLESS_BASE/api/documents/post_document/"

Returns task UUID. Verify success via:

curl ... "$PAPERLESS_BASE/api/tasks/?task_id=<UUID>"

Batch upload sa klasifikacijom

Skripta: /tmp/paperless-classify-v2.py (commit u repo-u TBD)

python3 /tmp/paperless-classify-v2.py --dry --all     # dry-run all ~/ALAI/*
python3 /tmp/paperless-classify-v2.py --all           # actual upload
python3 /tmp/paperless-classify-v2.py FILE [FILE...]  # specific files

Klasifikator mapira path → (storage_path, correspondent, document_type, tags) prema rules engine-u. Pre-upload dedup po normalized title; Paperless takođe ima vlastiti content-hash dedup (rejects file ako mu je sadržaj već prisutan).

Operations cheat sheet

# Document count
curl ... "$BASE/api/documents/?page_size=1" | jq '.count'

# Latest 10 docs
curl ... "$BASE/api/documents/?ordering=-created&page_size=10" | jq '.results[]|{id,title,created}'

# Search by tag
curl ... "$BASE/api/documents/?tags__id=17" | jq '.results[].title'

# Search by storage path
curl ... "$BASE/api/documents/?storage_path__id=1"

# Full-text search (OCR'd content)
curl ... "$BASE/api/documents/?query=finanstilsynet"

# Task queue status
curl ... "$BASE/api/tasks/?page_size=200" | jq 'group_by(.status)|map({status:.[0].status,count:length})'

# Failed tasks (often = content duplicates)
curl ... "$BASE/api/tasks/" | jq '[.[]|select(.status=="FAILURE")|{file:.task_file_name,reason:.result}]'

Architecture

[ALAI LAN egress 92.221.168.61]
       │
       ▼
[Cloudflare]
   ├─ IP Access Rule: bypass WAF (Layer 1)
   └─ CF Access policy: bypass Zero Trust (Layer 2)
       │
       ▼
[Caddy on Azure VM 4.223.110.181]
   archive.alai.no → paperless-ngx:8000
       │
       ▼
[alai-paperless-1 container]
   ├─ alai-paperless-redis-1 (queue)
   ├─ alai-paperless-gotenberg-1 (PDF preview)
   └─ alai-paperless-tika-1 (text extraction)
       │
       ▼
[Postgres + media volume on Azure VM]

Web login

CEO alembasic superuser created 2026-04-28. Initial password rotirana — koristi BW item ili lični password.

Pristup sa Mac Air (LAN egress 92.221.168.61, u CF Access bypass) → direktno na https://archive.alai.no bez CF SSO challenge. Login Paperless web UI sa username + password. Promijeni password kroz Profile → Change Password.

Iz Mac Studio (VPN aktivan) — backend dostupan ali samo via API sa bind interface, ne web browser (browser ne prima --interface flag).

Outstanding (TODO)

ALAI Contacts Inventory

ALAI Contacts Inventory

Authoritative source: Paperless-ngx Correspondents na archive.alai.no/api/correspondents/ Last rebuild: 2026-04-28 (iz ~/system/databases/email-inbox.db, 2299 emails) MC reference: #9546

Sve poslovne kontakte ALAI Holding AS i partnerskih kompanija. Auto-rebuild iz email DB-a + lokalnih dokumenata. Single source of truth.

Kako pretraživati kontakte

Web UI

https://archive.alai.no → Correspondents tab → search ili browse.

API

source ~/.config/alai/paperless-token.env
curl -s --interface 192.168.68.65 \
  -H "Authorization: Token $PAPERLESS_TOKEN" \
  "https://archive.alai.no/api/correspondents/?name__icontains=sintef" | jq

Stat — koliko ih je

curl ... "https://archive.alai.no/api/correspondents/?page_size=1" | jq '.count'

Trenutni inventory — 56 correspondents (2026-04-28)

Banking / fintech partneri & klijenti

ID Name Source
19 PBZ Zagreb seeded — Intesa pivot HR
20 Intesa Sanpaolo seeded
29 Vidar Aksland (SpareBank1 Sør-Norge) email
30 Tomislav Premuž (PBZ Zagreb) email
31 Vegard Aven (ZTLPay) email
32 Andreas Bjerke (ZTLPay) email
33 Aprila Bank ASA email
34 Folio email — Bilko reference

Regulatori / vlast

ID Name Source
16 Finanstilsynet seeded
17 Skatteetaten seeded
18 Brønnøysundregistrene seeded
— Innovasjon Norge seeded

Akademski / research partneri

ID Name Source
15 SINTEF seeded
26 Brian Elvesæter email — SINTEF
27 Signe Riemer-Sørensen email — SINTEF lead
28 Harald Rønn email — Simula
35 Håkon Kløve-Graue Lavik email — Finance Innovation Bergen

HR / recruiters / consultancies

ID Name
36 Kjell Ljøstad (Hive Consulting)
37 Audun (Kons AS)
38 Amanda Heie Veiby (Emagine)
39 Ove Olsen (Knowit)
40 Amila Lagumdžija (Authority Partners)
41 Elakkiya Sivakumar (Storebrand)
42 Henrik Digernes (NFF)
43 Thomas Dahlsrud (Sykling)

Network / kolege / familija

ID Name
44 Hamdija Salkić (LinkedIn)
45 Asmir Merdžanović
46 Anel Pasić (WizardNUF)
47 Adnan Cesko
48 Stefan (Smitrovic)
49 Emma Hu (Transtek)
50 MARFILD HOLD

Vendors sa account managerom

ID Name
21 Anthropic
22 Cloudflare
23 Tryg
24 Fiken AS
25 Entur AS
51 Knut at Sanity
52 Dan at Vercel
53 Sanity.io
54 Kravia
55 Vercel Security
56 Tryg Forsikring

Kako auto-update-ovati

Svaka Claude sesija (per ZAKON ARCHIVE FIRST):

# ~/system/scripts/contacts-rebuild.py (TODO — kreirat će ga FlowForge u sljedećoj iteraciji)
import sqlite3, requests
db = sqlite3.connect("~/system/databases/email-inbox.db")
new_senders = db.execute("""
  SELECT from_addr, from_name, COUNT(*) c
  FROM emails
  WHERE classification != 'SPAM' AND from_name != ''
    AND from_addr NOT LIKE '%no-reply%'
    AND from_addr NOT LIKE '%newsletter%'
  GROUP BY from_addr HAVING c >= 2
""")
# Compare with existing Paperless correspondents
# Auto-create new ones with name + meta in notes

Trigger:

Multi-tenant kontekst (Bilko HR/BiH/Srbija)

archive.alai.no postaje SaaS feature kroz Bilko klijente. Trenutni stanje = single instance, single tenant. Future:

Dok to nije implementirano, sve ide pod /ALAI/contacts/ storage path (id=10).

Outstanding TODO

MC Quality Trail — validator_agent + agent_output write semantics (Patch #10036)

Summary

Patch bundle MC #10036 (landed 2026-04-29) backfills two missing quality-trail columns in Mission Control. Prior to this patch, 0 of 6,529 tasks had validator_agent populated, making audit and quality trending impossible. The patch adds --validator <slug> and --quality <int> flags to mc.js ready, and makes mc.js done write the task outcome to agent_output when that column is NULL. All writes use no-clobber semantics so existing data is never overwritten. Validated by Proveo (MC #10038, 8/8 PASS, GLOBAL_VERDICT: PASS).

New CLI flags — mc.js ready

FlagTypeValidationEffect
--validator <slug>stringregex [a-z][a-z0-9_-]{1,40}Writes validator_agent + validation_timestamp = datetime('now')
--quality <int>integer0–10 inclusiveWrites quality_score

No-clobber semantics

Both flags are strictly optional. If a flag is absent or empty the corresponding column is not touched — existing values are preserved. This applies to both commands:

Postflight derivation — GLOBAL_VERDICT to quality score

The task-postflight skill (SKILL.md Section 6) derives the --quality integer from the GLOBAL_VERDICT emitted by the validator agent:

GLOBAL_VERDICT--quality value
PASS10
PARTIAL5
FAIL0

Example invocation

node ~/system/tools/mc.js ready 9999 "validation passed" --validator proveo --quality 10

This single command marks the task ready, records the validator identity, timestamps the validation, and writes the quality score in one atomic call.

Audit query

To find tasks closed after the patch date that are still missing a validator (quality trail gap):

SELECT id, title, status, completed_at FROM tasks
WHERE status='done' AND validator_agent IS NULL AND completed_at > '2026-04-29';

Cross-references

pi-orchestrator: interactive_protection — skip_owners + grace window (Patch #10063)

Summary

Symptom: John's H/M tasks added via mc.js add were being auto-paused by pi-orchestrator within seconds, before manual orchestration (prompt-forge → mehanik → dispatch pipeline) could begin.
Root cause: The daemon's getNextTask() loop claimed any open H/M task with no owner-based exclusion, colliding with John's interactive workflow where tasks are created as open before the human pipeline starts.
Fix: Added interactive_protection config block (skip_owners + grace_seconds) and updated the SQL WHERE clause in pi-orchestrator.js (lines 3412–3471) to respect both guards.
Date: 2026-04-29 | Patch: MC #10063 | Proveo verdict: 8/8 PASS

Root Cause Detail

Prior to this patch, getNextTask() in ~/system/kernel/pi-orchestrator.js selected tasks with a WHERE clause limited to status and priority, with no consideration for task owner. This meant:

The collision was systematic: mc.js add happens before manual orchestration begins by design (ZAKON #25). The daemon treated all open tasks as automation targets.

New Config Keys

File: ~/system/config/pi-orchestrator-config.json (block added at patch time):

"interactive_protection": {
  "skip_owners": ["john"],
  "grace_seconds": 300
}

Pickup Logic Table

OwnerAgePicked up?Reason
johnanyNOInteractive owner — skip_owners guard
NULL (unowned)< 300sNOGrace window — task may still be in orchestration queue
NULL (unowned)≥ 300sYESTrue unassigned automation task
pi-orchestrator / autowork / agent-workeranyYESExplicit automation owners — allowlist passthrough
edita / other non-protectedanyYESExisting behavior unchanged

Verification Commands

# Confirm daemon is running with new code
launchctl list | grep pi-orchestrator   # should return active PID (e.g. 31020)

# Confirm config has interactive_protection block
cat ~/system/config/pi-orchestrator-config.json | jq .interactive_protection

# Live test: create H task as john, verify NOT auto-paused after 90s
ID=$(node ~/system/tools/mc.js add "VERIFY-protection" --priority H --owner john --category system | grep -oE '#[0-9]+' | tr -d '#')
sleep 90
node ~/system/tools/mc.js show $ID | grep -E "Status|Owner"
# Expected: Status = open  |  Owner = john

Daemon Reload Procedure

Run this after any change to pi-orchestrator-config.json or pi-orchestrator.js:

launchctl unload ~/system/daemons/launchagents/com.john.pi-orchestrator.plist
launchctl load  ~/system/daemons/launchagents/com.john.pi-orchestrator.plist
sleep 5 && launchctl list | grep pi-orchestrator   # PID should be present

Note: The daemon reads config at startup. Changes to interactive_protection are not hot-reloaded; a full unload/load cycle is required.

Tuning Guide

Adding another interactive owner (e.g. alem)

Edit ~/system/config/pi-orchestrator-config.json: add the owner name to the skip_owners array, save, then reload the daemon (procedure above).

Extending the grace window for slower workflows

Edit ~/system/config/pi-orchestrator-config.json: increase grace_seconds (e.g. 600 for 10-minute window), save, then reload the daemon.

Verifying config was picked up after reload

grep -i "interactive_protection|skip_owners|grace" ~/system/logs/pi-orchestrator.log | tail -5

Evidence & Cross-References

MLX Router — Local Inference Gateway

MLX Router — Local Inference Gateway

MLX Router — Local Inference Gateway

بِسْمِ ٱللَّهِ ٱلرَّحْمَـٰنِ ٱلرَّحِيمِ

Service: mlx-router (com.alai.mlx-router) Port: 11500 (127.0.0.1) Status: Production (2026-05-01) Owner: ALAI System Infrastructure MC: #10429

Overview

MLX Router is ALAI’s local inference gateway that routes AI inference requests to zero-cost MLX models running on ANVIL (Mac Studio M3 Ultra, 96GB). It provides tier-fallback routing: tier1 MLX (local, $0 cost) → tier2 FORGE Ollama ($0 cost) → tier3 Anthropic API (metered cost).

Purpose: Reduce inference costs by offloading read-only agent workloads to local MLX models. Anthropic API is reserved for tier3 fallback only.

Cost Savings: All MLX and FORGE requests logged at cost_usd=0. Estimated 95%+ reduction in inference costs for wired agents.

Architecture

flowchart LR
    subgraph Caller
        A[Agent/Client]
    end
    
    subgraph MLX-Router["mlx-router.js :11500"]
        R[Route by model_class]
    end
    
    subgraph Tier1["Tier 1: MLX Local (ANVIL)"]
        M1[classify → :11437<br/>Qwen3-8B-4bit]
        M2[code → :11438<br/>Qwen2.5-Coder-32B]
        M3[reason → :11435<br/>Gemma-4-26B]
        M4[audit → :11436<br/>Qwen3-32B]
    end
    
    subgraph Tier2["Tier 2: FORGE Ollama"]
        F[10.0.0.2:11434<br/>qwen3/deepseek/etc]
    end
    
    subgraph Tier3["Tier 3: Anthropic API"]
        C[claude-haiku/sonnet/opus]
    end
    
    A -->|POST /v1/chat| R
    R -->|Health: UP| M1
    R -->|Health: UP| M2
    R -->|Health: UP| M3
    R -->|Health: UP| M4
    R -->|Tier1 DOWN| F
    F -->|Tier2 FAIL| C
    
    M1 -.->|cost_usd=0| CT[cost-tracker.js]
    M2 -.->|cost_usd=0| CT
    M3 -.->|cost_usd=0| CT
    M4 -.->|cost_usd=0| CT
    F -.->|cost_usd=0| CT
    C -.->|metered| CT

Service Management

Start

launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.alai.mlx-router.plist

Stop

launchctl bootout gui/$(id -u)/com.alai.mlx-router

Restart

launchctl bootout gui/$(id -u)/com.alai.mlx-router
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.alai.mlx-router.plist

Check Status

launchctl print gui/$(id -u)/com.alai.mlx-router
# Look for: state = running
# Get PID from output

View Logs

# Stdout (health probes, routing decisions)
tail -f /tmp/com.alai.mlx-router.stdout.log

# Stderr (errors)
tail -f /tmp/com.alai.mlx-router.stderr.log

Health Check

Endpoint

curl -s http://127.0.0.1:11500/health | jq

Expected Output

{
  "status": "ok",
  "endpoints": {
    "classify": { "available": true, "lastCheck": "2026-05-01T09:00:00.000Z", "latencyMs": 8 },
    "code":     { "available": true, "lastCheck": "2026-05-01T09:00:00.000Z", "latencyMs": 7 },
    "reason":   { "available": true, "lastCheck": "2026-05-01T09:00:00.000Z", "latencyMs": 3 },
    "audit":    { "available": true, "lastCheck": "2026-05-01T09:00:00.000Z", "latencyMs": 5 }
  }
}

Healthy state: All four endpoints show available: true, latency <50ms.

Unhealthy state: If available: false, that model_class will fall to tier2 FORGE on next request.

Model Classes

model_class Port Model RAM (GB) Use Case Latency
classify 11437 Qwen3-8B-4bit 5 Classification, routing, QA ~14s
code 11438 Qwen2.5-Coder-32B-Instruct 19 Code generation, review ~117s
reason 11435 Gemma-4-26B (MoE 4B active) 15 Reasoning, synthesis, validation ~94s
audit 11436 Qwen3-32B-4bit 17 Architecture audit, analysis ~120s (est)

Note on latency: MLX inference is sequential and slow. 8B models take ~14s, 32B models take ~94-117s. Not suitable for synchronous user-facing work. Use for background/async agent tasks only.

Tier Fallback Chain

  1. Tier 1 — MLX Local (ANVIL): 127.0.0.1 ports 11435-11438, cost=$0
    • Health-gated: If endpoint available: false, skip to tier2
    • Timeout: 120s
  2. Tier 2 — FORGE Ollama: 10.0.0.2:11434, cost=$0
    • Models: qwen3:8b (classify), qwen3-coder:latest (code), deepseek-r1:70b (reason), qwen3:32b (audit)
    • Timeout: 60s
  3. Tier 3 — Anthropic API: Metered cost
    • Models: claude-haiku-4-5 (classify), claude-sonnet-4-6 (code/reason), claude-opus-4-7 (audit)
    • Timeout: 60s

Fallback triggers: HTTP error, timeout, or health probe failure. Router tries tier1 → tier2 → tier3 until success or exhaustion.

Cost Verification

All MLX and FORGE requests log cost_usd=0 to cost-tracker.js.

# Check today's MLX costs (should be 0.0)
node ~/system/tools/cost-tracker.js summary today | grep mlx-local

# Query cost_events.db directly
sqlite3 ~/system/databases/costs.db \
  "SELECT backend, SUM(cost_usd) as total, COUNT(*) as requests 
   FROM cost_events 
   WHERE backend='mlx-local' 
   GROUP BY backend;"

# Expected: mlx-local | 0.0 | <count>

Adding a New Model Class

  1. Add MLX endpoint to ~/system/tools/mlx-router.js in MLX_ENDPOINTS object:

    new_class: {
  url: 'http://127.0.0.1:11439',
  modelId: '/Users/makinja/system/research/mlx-models/NewModel-4bit',
  shortname: 'newmodel-mlx',
  maxConcurrent: 1,
}

  2. Add tier2 FORGE fallback in FORGE_FALLBACK:

    new_class: { model: 'forge-model:latest', url: 'http://10.0.0.2:11434' }

  3. Add tier3 Anthropic fallback in ANTHROPIC_FALLBACK:

    new_class: 'claude-sonnet-4-6'

  4. Extend capability table at ~/system/specs/mlx-capability-table.md with routing rationale.

  5. Restart daemon:

    launchctl bootout gui/$(id -u)/com.alai.mlx-router
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.alai.mlx-router.plist

  6. Verify health:

    curl -s http://127.0.0.1:11500/health | jq '.endpoints.new_class'
# Should show available: true

Wiring an Agent

Add inference: block to agent’s YAML frontmatter in ~/system/agents/definitions/<agent>.md:

inference:
  prefer_inference: mlx-router
  model_class: classify
  router_url: http://127.0.0.1:11500/v1/chat
  rationale: "Read-only classification tasks — no production risk"
  wired_by: skillforge/MC#<id>/<date>

Sync to active agents:

~/bin/agent-definitions-sync.sh

Currently wired agents (2026-05-01): - sentinel-tester (classify) - sentinel-validator (reason) - sentinel-architect (audit)

Failure Modes

Failure Symptom Impact Mitigation
MLX daemon down Health probe shows available: false Falls to tier2 FORGE Automatic failover; check LaunchAgent logs
FORGE down Tier2 request fails Falls to tier3 Anthropic Cost increase; alert if sustained
All MLX endpoints down All classes fall to tier2/tier3 Full Anthropic cost Restart MLX daemons (4 LaunchAgents on ANVIL)
mlx-router daemon down No service on :11500 Agent inference fails Restart com.alai.mlx-router LaunchAgent
Timeout (8B model >120s) Request slow/stuck Falls to tier2 Normal for large prompts; reduce max_tokens

Performance Expectations

Latency (tier1 MLX): - 8B classify: ~14s (measured) - 32B code: ~117s (measured) - 32B reason: ~94s (measured) - 32B audit: ~120s (estimated)

Concurrency: - classify: 2 parallel requests - code/reason/audit: 1 request at a time (MLX is sequential)

Not for: - User-facing synchronous requests (too slow) - Real-time classification (<1s SLA)

Good for: - Background agent tasks (sentinel audit, QA checks) - Async workflows (overnight batch processing) - Read-only analysis (no Write/Edit risk)

Logs & Debugging

Daemon logs:

# Health probe output every 60s
tail -f /tmp/com.alai.mlx-router.stdout.log

# Example healthy output:
# [mlx-router] Health probe:
#   classify: UP (8ms)
#   code: UP (7ms)
#   reason: UP (3ms)
#   audit: UP (5ms)

Request routing logs:

# Each request logs tier used
# Example: [mlx-router] tier1 classify → qwen3-8b-mlx (470ms)
# Tier2/tier3 fallback logged with reason

Cost tracking:

# Every request creates a cost_events entry
sqlite3 ~/system/databases/costs.db \
  "SELECT model, backend, cost_usd, timestamp 
   FROM cost_events 
   WHERE backend='mlx-local' 
   ORDER BY timestamp DESC 
   LIMIT 10;"

MC History

Last Updated: 2026-05-01 Status: Production Validation: Proveo 10/10 PASS

active-thread-lock — 4th Anti-Drift Structural Layer

1. TL;DR

active-thread-lock.sh is a PreToolUse hook that fires on every Task, Agent, WebSearch, and WebFetch tool call. It reads the ## ACTIVE_THREAD: block from ~/.claude/session-state.md, extracts the approved MC IDs (5-digit #NNNNN references), and blocks any dispatch whose prompt references an MC ID outside that approved set with exit code 2. To perform a legitimate CEO-authorized thread switch, include the token [CEO_APPROVED_THREAD_SWITCH] anywhere in the dispatch prompt. On any parse failure or missing configuration the hook exits 0 (fail-open), so it never blocks legitimate non-MC work.

2. Why This Exists (Genesis)

ZAKON #27 (one product per session) has existed as a written rule since the ALAI operating system was established, but had no machine enforcement. The consequence was documented in feedback_drift_after_step1_completion.md (2026-05-02): John completed Step 1 of a CEO multi-step sequence, then drifted to a self-ranked priority (Akershus) instead of proceeding to Step 2, requiring the CEO to manually correct course. The CEO observation was: "ja vise ne mogu da te stalno vracam" ("I cannot keep pulling you back").

The fix was approved as part of the system-uvezivanje master spec §4 (~/system/specs/system-uvezivanje-master-2026-05-02.md), which defines four anti-drift structural layers. This hook is layer 4. The specific CEO directive is recorded in ~/system/specs/ai-factory-pipeline.md §6 Q3 answer: "Da" (2026-05-03).

The Four Anti-Drift Layers (system-uvezivanje §4)

LayerHook / MechanismWhat it enforces
1john-max-depth-gate.sh (ZAKON #28)Emergent-spawn depth ≤ 3 beyond Mehanik clearance
2pre-mc-add-gate.sh1 CEO turn = max N MC dispatches
3memo-citation-gate.shDrift-stop protocol on feedback memo citations
4active-thread-lock.sh (this runbook)ACTIVE_THREAD sequence enforcement — blocks off-thread MC dispatches

3. How It Works

Execution Flow (Step-by-Step)

  1. Read JSON from stdin. Claude Code passes a JSON object with tool_name and tool_input fields. The hook extracts tool_input.prompt via Python.
  2. Extract dispatched MC IDs from prompt. Regex patterns matched (4-6 digit numbers):
    • MC #NNNNN or #NNNNN
    • mc_task_id NNNNN or task-id NNNNN
    If no MC IDs are found in the prompt, exit 0 (fail-open — no IDs to check).
  3. Check bypass token. If [CEO_APPROVED_THREAD_SWITCH] is present anywhere in the prompt, exit 0 (authorized override). This check fires before any file I/O.
  4. Read session-state.md. File: ~/.claude/session-state.md. If the file is missing, exit 0 (fail-open).
  5. Extract approved IDs from ACTIVE_THREAD block. Python regex finds the block starting at ## ACTIVE_THREAD:, continuing until the next --- separator or next ## [A-Z] heading. All #NNNNN patterns within that block form the approved set. If the block is absent or yields no IDs, exit 0 (fail-open).
  6. Compare. For each dispatched MC ID, check against the approved set. First non-member triggers exit 2 with a BLOCKED message to stderr naming the offending MC ID, the full approved set, and the override token. All members pass with exit 0.

Pseudocode

INPUT = read_stdin_json()
PROMPT = INPUT.tool_input.prompt

DISPATCHED = extract_mc_ids(PROMPT)
# Patterns: #NNNNN, MC #NNNNN, mc_task_id NNNNN, task-id NNNNN (4-6 digits)
if DISPATCHED is empty:
    exit 0  # no IDs -- fail-open

if "[CEO_APPROVED_THREAD_SWITCH]" in PROMPT:
    exit 0  # bypass token

if not exists("~/.claude/session-state.md"):
    exit 0  # missing file -- fail-open

APPROVED = extract_mc_ids_from_active_thread_block("~/.claude/session-state.md")
if APPROVED is empty:
    exit 0  # no block or malformed -- fail-open

for id in DISPATCHED:
    if id not in APPROVED:
        stderr("BLOCKED [active-thread-lock]: MC #" + id
               + " not in ACTIVE_THREAD sequence (approved set: "
               + join(APPROVED) + "). Override: include [CEO_APPROVED_THREAD_SWITCH] in prompt.")
        exit 2

exit 0

4. Override

Include the literal string [CEO_APPROVED_THREAD_SWITCH] anywhere in the dispatch prompt. The hook checks for this token before reading session-state.md, so it incurs no file I/O on bypass.

Use case: CEO explicitly authorizes work on an MC outside the current thread, e.g., an urgent hotfix on a separate product. The CEO must include or authorize this token in their directive — it cannot be inserted by John autonomously.

Important: Inserting [CEO_APPROVED_THREAD_SWITCH] without explicit CEO authorization is itself a drift violation tracked by the memo-citation gate (layer 3).

5. Fail-Open Conditions

The hook exits 0 (allow) under every condition below. It never produces false positives against legitimate non-MC dispatches.

ConditionExitStderr signal
No 5-digit MC ID extractable from prompt0(silent)
[CEO_APPROVED_THREAD_SWITCH] token present in prompt0(silent)
~/.claude/session-state.md does not exist0[active-thread-lock] session-state.md not found — fail-open.
session-state.md exists, no ## ACTIVE_THREAD: block found0[active-thread-lock] No ACTIVE_THREAD block or no MC IDs found in session-state.md — fail-open.
ACTIVE_THREAD block present but contains no parseable #NNNNN IDs0[active-thread-lock] No ACTIVE_THREAD block or no MC IDs found in session-state.md — fail-open.
Python internal exception during parse0[active-thread-lock] ACTIVE_THREAD block parse error — fail-open.

6. Smoke Test Procedure

Independent Proveo replay completed 2026-05-03. Evidence: /tmp/evidence-99014-proveo/replay-log.txt and verdict.txt. Overall verdict: 7/7 PASS.

To replay a single TC manually:

echo '{"tool_name":"Task","tool_input":{"prompt":"Dispatch codecraft agent to build MC #10612."}}' \
  | bash ~/.claude/hooks/active-thread-lock.sh
echo "Exit code: $?"
TCDescriptionFixture promptExpected exitExpected stderr signal
TC1MC is in ACTIVE_THREAD approved setDispatch codecraft agent to build MC #10612 system-uvezivanje hook.0(silent)
TC2MC is NOT in approved setDispatch flowforge agent to work on MC #99999 some unrelated task.2BLOCKED [active-thread-lock]: MC #99999 not in ACTIVE_THREAD sequence (approved set: 10424,10429,10536,10611,10612,99012,99013,99014,99015,99016). Override: include [CEO_APPROVED_THREAD_SWITCH] in prompt.
TC3session-state.md removed entirelyDispatch agent to work on MC #99999.0[active-thread-lock] session-state.md not found — fail-open.
TC4session-state.md present, no ACTIVE_THREAD blockDispatch agent to work on MC #99999.0[active-thread-lock] No ACTIVE_THREAD block or no MC IDs found in session-state.md — fail-open.
TC5[CEO_APPROVED_THREAD_SWITCH] token present + unapproved MC[CEO_APPROVED_THREAD_SWITCH] Dispatch agent to work on MC #99999 special task.0(silent)
TC6Prompt has no 5-digit MC ID at allDispatch agent to review the documentation and run tests.0(silent)
TC7ACTIVE_THREAD block present but contains no parseable #NNNNN IDsDispatch agent to work on MC #99999.0[active-thread-lock] No ACTIVE_THREAD block or no MC IDs found in session-state.md — fail-open.

7. How to Update ACTIVE_THREAD When Starting a New Master Thread

The hook reads ~/.claude/session-state.md fresh on every dispatch. No restart or cache clear is needed — edits take effect on the very next dispatch call.

Operational Procedure

  1. Open ~/.claude/session-state.md.
  2. Find or create the ## ACTIVE_THREAD: block at the top of the file, before any archived thread sections or --- separators.
  3. Write the block in the format below, listing every approved child MC ID using the #NNNNN pattern anywhere in the block (the hook scans the full block for all such patterns).
  4. Save. The hook picks up the new state automatically on the next dispatch.

Example Block Format (Actual from Current Session)

## ACTIVE_THREAD: system-uvezivanje-master (CEO approved 2026-05-02 23:55)

**Spec:** ~/system/specs/system-uvezivanje-master-2026-05-02.md
**Master MC:** #10612
**SEQUENCE:** B -> C -> A
**CURRENT_STEP:** B
**LAST_COMPLETED:** (none)

**Children (CEO answers 2026-05-03 to ai-factory-pipeline.md §6):**
1. #99012 [H] Blueprint-check Phase 3 build
2. #99013 [H] alai-hooks Kotlin source check-in
3. #99014 [H] active-thread-lock hook
4. #99015 [H] one-ceo-turn-mc-cap.sh counter fix
5. #99016 [H] Migrate duplicate bash gates to Kotlin

**DRIFT-STOP:** Any task outside ACTIVE_THREAD = STOP, write memo, ask CEO.
Override = explicit CEO [CEO_APPROVED_THREAD_SWITCH] token in CEO message.

Adding a new approved MC mid-session: Append a line with #NNNNN to the block. The hook includes it on the next dispatch.

Closing a thread: Archive the block by moving it below a --- separator or rename the heading to ## ARCHIVED:. With no active ## ACTIVE_THREAD: block, the hook is fully fail-open and imposes no constraint.

8. Wiring

Position in settings.json

File: ~/.claude/settings.json. Event: PreToolUse. Matcher: Task|Agent|WebSearch|WebFetch. The hook is at index position 4 (0-indexed) within the matcher block, sitting after pre-dispatch-gate.sh (index 3) and before john-max-depth-gate.sh (index 5).

PreToolUse — matcher: Task|Agent|WebSearch|WebFetch
  [0] bash ~/.claude/hooks/lock-john-dispatch-cap.sh
  [1] ~/.claude/hooks/claude-hooks pre         (Kotlin alai-hooks binary)
  [2] bash ~/.claude/hooks/pre-action-da-gate.sh
  [3] bash ~/.claude/hooks/pre-dispatch-gate.sh
  [4] bash ~/.claude/hooks/active-thread-lock.sh   <-- THIS HOOK
  [5] bash ~/.claude/hooks/john-max-depth-gate.sh
  [6] bash ~/.claude/hooks/one-ceo-turn-dispatch-cap.sh

Hook Artifact Details

FieldValue
Path~/.claude/hooks/active-thread-lock.sh
Size3984 bytes (104 lines)
sha256e3c7ce8b8b1cb45968e368a4e7872df923f1af97a37303296ecb5cf28bf6fb79
LanguageBash + inline Python 3 (consistent with all other ALAI hooks)
Activated2026-05-03

9. Genesis MC + Commits

MC Grandfather Clause — Legacy Task Completion

MC Grandfather Clause — Legacy Task Completion

Installed: 2026-05-03 17:30 UTC
Implementation: /Users/makinja/system/tools/mc.js
Audit Log: /tmp/mc-grandfathered-completions.log
MC Genesis: #99057
Verified by: Proveo (angie-jones) — 6/6 PASS

What Is This?

The grandfather clause allows legacy tasks (created and marked ready_for_review before 2026-05-03 17:30 UTC) to complete without triggering the new gate stack:

Old gates still apply: ZAKON #22 ready_for_review requirement, evidence bundle, postflight marker, ADR-021 compliance. This is not a full bypass — only the new stack is skipped.

Why?

CEO decision 2026-05-03 after the 3-layer gate chain ('sistem koje je krc') blocked routine closure of 6 drift-prevention MCs that had already been validated under the old regime. The grandfather clause creates a clean boundary: tasks already validated = let them close without new gates; new tasks = all gates active.

How It Works

Timestamp Boundary

GATE_INSTALL_DATE = '2026-05-03T17:30:00Z' (line ~50 in mc.js)

The isGrandfathered(task, taskId) helper checks three timestamp heuristics (in priority order):

  1. completed_at (if task already done — should never happen in preCompletion gate)
  2. updated_at when status = ready_for_review
  3. Earliest task_history entry with action = READY_FOR_REVIEW

If any timestamp is before GATE_INSTALL_DATE → grandfathered = true.

UTC fix: SQLite timestamps are space-separated (YYYY-MM-DD HH:MM:SS) — JavaScript new Date() parses these as local time unless 'Z' is appended. The sqliteUtcMs() helper (lines 656-661) imposes UTC parsing to prevent timezone drift.

Gate Wrap Points

Five checks in preCompletionGate() (lines 750-817) are wrapped with if (grandfathered) { /* skip */ }:

Grandfathered vs Forced

Aspect GRANDFATHERED FORCED_COMPLETION
Trigger ready_for_review_at < gate install date override flag
Gates bypassed New stack only (qa-19, GOTCHA, hop-build, validator, trust labels) All gates (including old: ZAKON #22, evidence, postflight, ADR-021)
Audit log /tmp/mc-grandfathered-completions.log /tmp/mc-forced-completions.log
DB action task_history.action='GRANDFATHERED' task_history.action='FORCED_COMPLETION'
Console output "GRANDFATHERED: task predates new gate stack" "⚠️ FORCED COMPLETION"
Intended use Clean legacy backlog closure (automatic) CEO override for exceptional cases (explicit)

Audit Logs

Grandfathered Log

/tmp/mc-grandfathered-completions.log (separate from FORCED log)

Format:

{
  "timestamp": "2026-05-03T18:27:04.123Z",
  "taskId": 99022,
  "completionType": "GRANDFATHERED",
  "gate_install_date": "2026-05-03T17:30:00Z",
  "reason": "updated_at=2026-05-03 09:52:57 (status=ready_for_review)"
}

Database

Query:

sqlite3 ~/system/databases/mission-control.db \
  "SELECT task_id, action, timestamp FROM task_history WHERE action='GRANDFATHERED'"

Verified entries (as of 2026-05-03): 5 tasks (99022, 10590, 10608, 10611, 10613)

Code References

Element Location (mc.js)
GATE_INSTALL_DATE constant Line ~50
sqliteUtcMs() helper Lines 656-661
isGrandfathered() helper Lines 662-712
preCompletionGate() wrapper checks Lines 750-817
completionType assignment Lines 2030-2033
Audit log write Line ~2063

Verification

Test 1 (legacy task): MC #99022 (ready_for_review_at = 2026-05-03 08:43 UTC)
node ~/system/tools/mc.js done 99022
→ Status = done, action = GRANDFATHERED (no override needed)

Test 2 (new task): MC #99058 (created 2026-05-03 18:27 UTC)
isGrandfathered() returns { grandfathered: false }
→ All gates remain active

Proveo report: /tmp/postflight-99057/proveo-report.md — 6/6 AC PASS

Created: 2026-05-03
Owner: John (AI Director)
Company: ALAI Holding AS

Atomic-write pattern for shared state files (POSIX os.replace)

Atomic-Write Pattern for Shared State Files (POSIX os.replace)

1. Why This Matters

In a multi-session environment where hooks, tools, and daemons write to shared state files (JSON configs, task markers, session identifiers), a naive open() + write() + close() pattern creates a torn-write hazard:

Impact: ZAKON #27 (active-thread enforcement) and ZAKON #28 (max-depth gate) rely on per-session state files that must NEVER contain partial writes. A torn write to /tmp/mc-active-task-$PID causes the hook to fall back to the global /tmp/mc-active-task, silently defeating session isolation.

2. The Pattern — POSIX Atomic Rename

2.1 Python Pattern

The correct pattern uses tempfile + fsync + os.replace() to guarantee atomicity:

import os
import tempfile

def write_active_task(task_id, claude_pid=None):
    """Write active task for this session (atomic POSIX rename pattern).

    Writes to a tempfile in the same directory as the target, then uses
    os.replace() for an atomic swap. A crash or SIGKILL during the write
    leaves the target either absent (first write) or containing the previous
    complete value — never a partial write.
    """
    task_file = get_session_task_file(claude_pid)
    dir_ = os.path.dirname(task_file) or "."
    fd, tmp = tempfile.mkstemp(prefix=".active-task-", dir=dir_)
    try:
        with os.fdopen(fd, "w") as f:
            f.write(str(task_id))
            f.flush()
            os.fsync(f.fileno())
        os.replace(tmp, task_file)
    except Exception:
        try:
            os.unlink(tmp)
        except OSError:
            pass
        raise

Why this works:

  1. tempfile.mkstemp() creates a unique temp file in the SAME directory (same filesystem) as the target
  2. Write content to the temp file, flush buffers, call fsync() to ensure data is on disk
  3. os.replace(tmp, target) performs an atomic rename — POSIX guarantees this is a single syscall
  4. Readers see either the old complete file OR the new complete file — never a partial write
  5. If the process crashes before os.replace(), the temp file is abandoned but the target is untouched (or absent if first write)

2.2 Bash Pattern

For bash hooks writing to state files, use mktemp + mv pattern:

# Atomic write in bash using mktemp + mv
TARGET="/tmp/some-state-file.json"
CONTENT='{"count":0,"ts":"2026-05-03T10:00:00Z"}'

# Create temp file in same directory as target (same filesystem requirement)
TMP=$(mktemp "${TARGET}.XXXXXX")
echo "$CONTENT" > "$TMP"
mv -f "$TMP" "$TARGET"  # POSIX atomic on same filesystem

Why mv is atomic: On POSIX, mv within the same filesystem calls rename(2), which is atomic. Same guarantee as Python's os.replace().

Constraints:

3. What It Replaces — The Anti-Pattern

3.1 Python Anti-Pattern

DO NOT USE:

# WRONG — non-atomic, torn-write hazard
def write_active_task_WRONG(task_id, task_file):
    with open(task_file, "w") as f:
        f.write(str(task_id))

Why this is broken:

3.2 Bash Anti-Pattern

DO NOT USE:

# WRONG — torn-write hazard in bash
echo "$TASK_ID" > /tmp/mc-active-task-$$

The > operator truncates the file immediately, then writes. A crash between truncate and write completion leaves a zero-byte or partial file — identical hazard to the Python anti-pattern.

4. Same-Filesystem Requirement

The dir= kwarg in tempfile.mkstemp(prefix=".active-task-", dir=dir_) is critical:

Verification: df -h /tmp vs df -h ~/.claude/hooks — if different mount points, you MUST use dir= kwarg with target's parent directory.

For bash: Use mktemp "${TARGET}.XXXXXX" template — the suffix pattern ensures temp file is created in the same directory as $TARGET.

5. Crash Recovery Semantics

Scenario Before os.replace() After os.replace()
First write, no prior file Target absent, temp exists Target exists with new content
Overwrite existing file Target has old content, temp exists Target has new content
Crash during write() Target unchanged (or absent), temp partial/incomplete N/A — replace() never called
Crash during fsync() Target unchanged, temp may have partial data on disk N/A
Crash after os.replace() N/A Target has new complete content (atomic swap already done)

Key guarantee: The target file NEVER contains partial writes. A reader always sees either:

  1. File absent (no write has completed yet), OR
  2. File with the last successfully-completed write's full content

The exception handler (except: os.unlink(tmp)) cleans up the temp file on failure, preventing temp-file accumulation.

6. Testing Pattern

Unit test crash-recovery by mocking the write to raise an exception:

import unittest
import os
import tempfile
from unittest.mock import patch, mock_open

class TestAtomicWrite(unittest.TestCase):

    def test_crash_during_overwrite_preserves_old_content(self):
        """If write crashes after target exists, old content is preserved."""
        with tempfile.TemporaryDirectory() as tmpdir:
            target = os.path.join(tmpdir, "test-task.txt")

            # Write initial content
            with open(target, "w") as f:
                f.write("OLD-TASK-11111")

            # Simulate crash during second write
            with patch("builtins.open", side_effect=IOError("Simulated crash")):
                with self.assertRaises(IOError):
                    write_active_task_atomic("NEW-TASK-22222", target)

            # Old content must survive
            with open(target, "r") as f:
                content = f.read()
            self.assertEqual(content, "OLD-TASK-11111")

            # No temp files leaked
            leaked_temps = [f for f in os.listdir(tmpdir) if f.startswith(".active-task-")]
            self.assertEqual(len(leaked_temps), 0)

What this validates:

7. When to Apply

Use this pattern for any hook/lib writing JSON or state files where torn writes = corruption:

Do NOT use for:

8. Sites Covered

This pattern has been applied to the following high-risk state file writes:

8.1 Python Sites (Phase 2A — MC #99076)

8.2 Bash Hook Sites (Phase 2B-2 — MC #99080)

8 atomic-write patches applied across 4 hooks covering surfaces S3, S8, S9, S10:

File Line Pattern Surface Description
mc-turn-reset.sh 12 Python tempfile.mkstemp + os.replace S8 Reset MC turn counter
mc-turn-reset.sh 20 Bash mktemp + mv S3 Reset CEO_APPROVED token counter
mc-turn-reset.sh 23 Bash mktemp + mv S9 Reset dispatch turn counter
ceo-intent-classifier.sh 38 Python tempfile.mkstemp + os.replace S10 Write CEO intent classification
one-ceo-turn-dispatch-cap.sh 33 Python tempfile.mkstemp + os.replace S9 Increment dispatch counter
one-ceo-turn-dispatch-cap.sh 50 Python tempfile.mkstemp + os.replace S9 Rollback dispatch counter on failure
one-ceo-turn-mc-cap.sh 40 Python tempfile.mkstemp + os.replace S8 Increment MC add counter
one-ceo-turn-mc-cap.sh 59 Python tempfile.mkstemp + os.replace S8 Rollback MC counter on failure

Validation: All 8 sites passed Proveo crash-safety testing (AC5: runtime exception AFTER write+fsync but BEFORE os.replace/mv — old content preserved, no temp file leak). See /tmp/proveo-99080-2026-05-03.json.

8.3 Shadow-File Pattern for Human-Editable Shared State (Phase 2D — MC #99084)

For human-readable source files that must remain unmodified by automation (e.g., ~/.claude/session-state.md) but where enforcement hooks need per-session isolation, Phase 2D introduced the shadow-file pattern:

When to Use Shadow Files

The Shadow-File Pattern

Write a per-session machine-readable shadow file at /tmp/<key>-${SESSION_ID}.txt (atomically via mktemp+mv) at the same point the human-readable source is updated. Enforcement hooks read shadow-first with fallback to the human-readable source.

# Shadow write (in user-message-logger.sh at UserPromptSubmit)
# SESSION_ID resolution: stdin JSON → env CLAUDE_SESSION_ID → pid-$$ → REJECT (never "default")
_SHADOW_SESSION_ID="$SESSION_ID"
if [[ -z "$_SHADOW_SESSION_ID" ]]; then
    _SHADOW_SESSION_ID="${CLAUDE_SESSION_ID:-}"
fi
if [[ -z "$_SHADOW_SESSION_ID" ]]; then
    _SHADOW_SESSION_ID="pid-$$"
fi

_SHADOW_TARGET="/tmp/active-thread-${_SHADOW_SESSION_ID}.txt"
_SESSION_STATE_FILE="$HOME/.claude/session-state.md"

# Extract ACTIVE_THREAD IDs from session-state.md
_ACTIVE_THREAD_VALUE=$(python3 -c "
import re, sys
with open('$_SESSION_STATE_FILE', 'r') as f:
    content = f.read()
match = re.search(r'## ACTIVE_THREAD:.*?(?=\n---|\n## [A-Z]|\Z)', content, re.DOTALL)
if not match:
    sys.exit(1)
block = match.group(0)
ids = re.findall(r'#(\d{4,6})', block)
print('\n'.join(sorted(set(ids))))
" 2>/dev/null)

if [[ -n "$_ACTIVE_THREAD_VALUE" ]]; then
    # Atomic write: mktemp + mv
    _SHADOW_TMP=$(mktemp "${_SHADOW_TARGET}.XXXXXX")
    printf '%s\n' "$_ACTIVE_THREAD_VALUE" > "$_SHADOW_TMP"
    mv -f "$_SHADOW_TMP" "$_SHADOW_TARGET"
fi

# Shadow-first read (in active-thread-lock.sh)
_SHADOW_PATH="/tmp/active-thread-${SESSION_ID}.txt"
APPROVED_IDS=""

if [[ -f "$_SHADOW_PATH" ]]; then
    # Shadow file present: read per-session ACTIVE_THREAD (atomic, no stale-read risk)
    APPROVED_IDS=$(cat "$_SHADOW_PATH" 2>/dev/null || echo "")
else
    # Fallback: read session-state.md (global, backward-compatible)
    if [[ ! -f "$SESSION_STATE" ]]; then
        echo "[active-thread-lock] session-state.md not found and no shadow file — fail-open." >&2
        exit 0
    fi

    APPROVED_IDS=$(python3 -c "
import re, sys
with open('$SESSION_STATE', 'r') as f:
    content = f.read()
match = re.search(r'## ACTIVE_THREAD:.*?(?=\n---|\n## [A-Z]|\Z)', content, re.DOTALL)
if match:
    block = match.group(0)
    ids = re.findall(r'#(\d{4,6})', block)
    print('\n'.join(sorted(set(ids))))
" 2>/dev/null)
fi

Properties

Shadow-File Sites

Validation: Proveo PASS (6/6 ACs) — concurrent sessions with distinct session_id values read their own shadow files with no cross-session leak. Sessions without shadow files fall back to session-state.md with identical enforcement behavior. No "default" terminal value. See /tmp/proveo-99084-2026-05-03.json.

9. Reference

10. Further Reading

Generated by Skillforge for MC #99076 — Phase 2A Session Isolation Fix
Updated: 2026-05-03 (MC #99080 — Phase 2B-2 bash hook atomicity expansion)
Updated: 2026-05-03 (MC #99084 — Phase 2D shadow-file pattern for human-editable shared state)
Last verified: 2026-05-03 — Proveo Phase 2D report (PASS 6/6)

Spawn Gate Node-Side Parity (MC #10548)

Spawn Gate Node-Side Parity (MC #10548)

Context — Why This Exists

Problem: Pi-orchestrator spawns agents internally at Step 4.6 (~line 4291 in pi-orchestrator.js) without going through Claude Code's Task dispatch path. This meant PreToolUse Bash hooks (~/.claude/hooks/pre-dispatch-gate.sh) never fired, creating a bypass where internal spawns skipped Mehanik clearance verification.

Solution: MC #10548 implemented Node-side spawn gate parity — a JavaScript enforcement layer (~/system/kernel/spawn-gate.js) that mirrors all 9 checks from the Bash gate, called directly by pi-orchestrator before every agent spawn.

Genesis: Pi-orchestrator hardening Talas 2 (parent thread #10043 reform). Dependency on δ #10551 worktree_company_enforcer.js (completed).

Architecture — Dual Gate System

Gate Location Lines Trigger
Bash Gate ~/.claude/hooks/pre-dispatch-gate.sh 163 Claude Code Task/Agent dispatches (PreToolUse hook)
Node Gate ~/system/kernel/spawn-gate.js 853 Pi-orchestrator internal spawns (Step 4.6)

Shared enforcement: Both gates implement the same 9-check validation sequence. Checks 1-3 existed in spawn-gate.js before this MC; checks 4-9 were added to achieve full parity.

The 6 New Checks (MC #10548 Scope)

Check 4: Marker TTL (checkMarkerTTL, line 143)

Verifies Mehanik clearance has not expired. Reads expires_at field from /tmp/mehakin-cleared-{taskId} marker and compares to current time. Rejects if expired or missing.

Check 5: Marker Schema (checkMarkerSchema, line 175)

Validates all 22 required marker fields are present:

Check 6: Scope Ceiling (checkScopeCeiling, line 196)

Deterministic arithmetic enforcement: approved_subtask_count ≤ ceo_item_count + 2. Prevents scope creep beyond Mehanik-approved ceiling.

Check 7: Tool Contract (checkToolContract, line 226)

Research-class agent dispatches (datavera, sentinel-*) must include a TOOL_CONTRACT: block in the prompt. Exempts prompts with tool literal references (discover.js, lightrag.js, mc.js, web-search.sh) as implementation context, not research requests.

Fix on failure:

node ~/system/tools/wrap-with-tool-contract.js --agent research --tools web-search.sh

Check 8: Agent Registry (checkAgentRegistry, line 258)

Agent slug must exist in ~/system/agents/specialist-mapping.json. Bootstrap-exempt agents skip this check:

Check 9: Blueprint Advisory (checkBlueprintAdvisory, line 311)

WARN-ONLY, FAIL-OPEN. Checks blueprint_score < blueprint_threshold_applied. On failure, writes warning to /tmp/spawn-gate-warnings.log and stderr, but does NOT block dispatch. Bypassed by [CEO_OVERRIDE] in prompt.

Wiring — Pi-Orchestrator Integration

Location: ~/system/kernel/pi-orchestrator.js lines 4291-4316 (Step 4.6)

// Step 4.6: AAOS Spawn Gate — full 9-check parity with pre-dispatch-gate.sh (MC #10548)
if (_aaosSpawnGate && typeof _aaosSpawnGate.runGate === 'function') {
  const _taskPromptContext = `${task.title || ''} ${task.description || ''}`;
  const gateResult = await _aaosSpawnGate.runGate(task.id, 'pi-orchestrator', task.priority || 'M', _taskPromptContext);

  if (!gateResult.allowed) {
    const priority = (task.priority || 'M').toUpperCase();
    if (priority === 'H' || priority === 'M') {
      log('error', `Task #${task.id} SPAWN GATE BLOCKED: ${gateResult.reason}`, { taskId: task.id });
      execFileSync('node', [MC_SCRIPT, 'block', String(task.id), `spawn-gate: ${gateResult.reason}`], { timeout: 5000 });
      return; // HARD BLOCK — do not proceed to spawn
    }
    log('warn', `Task #${task.id} SPAWN GATE warned (L-priority): ${gateResult.reason}`, { taskId: task.id });
  }
}

On FAIL (H/M priority):

  1. Log error
  2. Call mc.js block <taskId> "spawn-gate: {reason}"
  3. Return early (no agent spawn)

On FAIL (L priority): Log warning, proceed with spawn.

Fallback: Lines 4316+ preserve legacy check() path for compatibility if runGate() is unavailable (pre-MC#10548 deployments).

Current State

Status: Code deployed COLD. Pi-orchestrator daemon is STOPPED.

Activation: Requires MC #10542 reactivation go/no-go decision (separate CEO approval, ZAKON PI2 verification pending).

Test Plan

Test suite: ~/system/tests/spawn-gate.test.js (23 tests)

Coverage: Each new check has PASS + FAIL path tests.

Run tests:

node --test ~/system/tests/spawn-gate.test.js

Expected output: 23 PASS, 0 FAIL

Validation

Proveo verdict: /tmp/proveo-10548-spawn-gate-validation.json — PASS 6/6 AC

SQLite DB Backup — Pillar #9 LITE

Genesis: Pillar #9 LITE (CEO approved 2026-05-05, DR-only scope). Spec: ~/system/specs/agentic-os-pillar9-LITE-2026-05-05.md. MC: #99248.

Daemon file: /Users/makinja/system/daemons/azure-db-backup.sh (524 lines)

Schedule: Every 4 hours via LaunchAgent com.alai.azure-db-backup (StartInterval=14400)

Overview

This runbook covers the SQLite backup phase of azure-db-backup.sh. It documents what gets backed up, how backups are produced, how to restore from Azure Blob Storage, and how to add new databases to the backup set.

The SQLite backup extension was introduced under Pillar #9 LITE to provide DR coverage for the four critical local SQLite databases that drive ALAI operational systems. Backups land in the same Azure Blob Storage container as Docker volume, Postgres, and Qdrant backups, under the sqlite/ prefix.

Blob lifecycle policy: 30 days Cool to Archive, deleted at 365 days (existing container policy, no additional configuration required).

Slack channel #ops receives an alert if 3 or more consecutive backup runs fail (MAX_FAILURES=3 in the script).

What Gets Backed Up

Four databases are covered, defined at lines 466-471 of the daemon file (SQLITE_DBS array):

LabelPathPurpose
mission-control$HOME/system/databases/mission-control.dbAll MC tasks, owners, priorities, status history
hivemind$HOME/system/databases/hivemind.dbInstitutional knowledge, facts, session summaries
costs$HOME/system/databases/costs.dbToken cost tracking, budget records
knowledge$HOME/system/databases/knowledge.dbExtracted knowledge index (187 MB)

The authoritative list lives in the SQLITE_DBS array at lines 466-471. Add new databases there; no other code change is required.

How It Works

The backup_sqlite() function (lines 420-462) executes four steps per database:

  1. Snapshot via online-backup API. sqlite3 .backup command is WAL-safe: it acquires a shared lock just long enough to copy pages, then releases it. The running application continues reading and writing throughout. Output: /tmp/alai-azure-backup-ts/sqlite-label-DATE.db
  2. Gzip compression. gzip -f replaces the snapshot file with .gz in-place.
  3. SHA-256 sidecar. The upload_blob() function computes a .sha256 file alongside the .gz before upload, enabling integrity verification at restore time.
  4. Azure Blob upload. az storage blob upload sends both the .gz and .sha256 sidecar to the container. Blob path pattern: sqlite/YYYY-MM-DD/label.db.gz

If the source database file is absent, the step is skipped with a WARN log entry (non-fatal). If sqlite3 .backup or gzip fails, the run is counted as a failure and the consecutive-failure counter increments.

Dry-run mode: Pass --dry-run to the script. No snapshot is taken, no upload occurs; the planned blob path is logged.

Restore Procedure

Restore target: vm-alai-support (4.223.110.181). SSH: ssh -i ~/.ssh/azure_alai alai-admin@4.223.110.181

Step 1 - Identify the backup to restore

List blobs: az storage blob list --account-name $AZURE_STORAGE_ACCOUNT --container-name $AZURE_CONTAINER_DB --prefix "sqlite/2026-05-05/" --auth-mode login --output table

Pick the label.db.gz blob you need.

Step 2 - Download and verify

Download the .db.gz blob and its .db.gz.sha256 sidecar to /tmp/restore/. Verify: sha256sum -c mission-control.db.gz.sha256

Step 3 - Decompress

gunzip /tmp/restore/mission-control.db.gz

Step 4 - Verify schema and data

Schema: sqlite3 /tmp/restore/mission-control.db ".schema"

Data: sqlite3 /tmp/restore/mission-control.db "SELECT id, title, status FROM tasks LIMIT 10;"

mc.js list should return task rows if the DB is valid.

Step 5 - Place into production path

Check for open handles: lsof | grep mission-control.db. Then: cp /tmp/restore/mission-control.db ~/system/databases/mission-control.db

Never overwrite a live database without passing the schema and data checks above.

Monitoring

Log file: ~/system/logs/azure-db-backup.log

Phase markers: grep "SQLite backup phase" ~/system/logs/azure-db-backup.log | tail -20

Per-DB success: grep "SQLite snapshot done" ~/system/logs/azure-db-backup.log | tail -20

Errors: grep -i "error.*sqlite\|warn.*sqlite" ~/system/logs/azure-db-backup.log | tail -20

Verify blobs for today: az storage blob list --account-name $AZURE_STORAGE_ACCOUNT --container-name $AZURE_CONTAINER_DB --prefix "sqlite/$(date +%Y-%m-%d)/" --output table

Expected: 8 blobs (4 x .db.gz + 4 x .db.gz.sha256) per successful run.

LaunchAgent health: launchctl list | grep azure-db-backup (PID non-zero = running; last column = exit code, 0 = success)

Adding a New DB

SQLITE_DBS array at lines 466-471 of /Users/makinja/system/daemons/azure-db-backup.sh is the single point of configuration.

  1. Open azure-db-backup.sh and locate the SQLITE_DBS array (lines 466-471).
  2. Append a new entry: "label:$HOME/system/databases/name.db". Use lowercase, hyphen-separated labels with no spaces.
  3. Test: bash ~/system/daemons/azure-db-backup.sh --dry-run 2>&1 | grep sqlite
  4. Confirm the planned blob path appears in log output for the new label.
  5. Update the "What Gets Backed Up" table in this runbook.
  6. Update MC #99248 or open a follow-up MC to track the addition.

No other code changes required. backup_sqlite() handles all databases uniformly.

Troubleshooting

SymptomLikely causeResolution
WARN: SQLite DB not found, skippingDatabase file does not exist at registered pathls -lh ~/system/databases/name.db. If path moved, update SQLITE_DBS array.
ERROR: sqlite3 .backup failedsqlite3 CLI missing, DB corrupted, or disk fullwhich sqlite3; df -h /tmp; sqlite3 name.db "PRAGMA integrity_check;"
ERROR: gzip failedDisk full on /tmpdf -h /tmp; rm -rf /tmp/alai-azure-backup-*
Blob missing after upload reports successSP credential expired or wrong containerVerify AZURE_CONTAINER_DB in ~/system/config/azure-backup.env.
Slack alert fires repeatedly3+ consecutive run failurestail -100 ~/system/logs/azure-db-backup.log. Fix phase. echo 0 > /tmp/azure-db-backup-failcount
LaunchAgent not running (PID=0)LaunchAgent unloaded or crashedlaunchctl load ~/Library/LaunchAgents/com.alai.azure-db-backup.plist

Telegram Bot Intent Classifier — comms-responder (#99290) — Intent classification fix for the ALAI Telegram bot. Same Operations Runbooks shelf.

Telegram Bot Intent Classifier — comms-responder

Telegram Bot Intent Classifier — comms-responder

MC: #99290, #99331 Status: Live Last deploy: 2026-05-05 22:47 UTC Code SHA-256: 233baff845b6c16153f900d1cab9756f84a72999f6425e375830a343b4870d36

1. Overview

The comms-responder is the intent-routing brain of the ALAI Telegram bot. It receives every inbound CEO message and classifies it into one of five intent categories before generating a response. Prior to MC #99290, the bot had a task-creation bias — informal messages such as "Bok" or "Kako si?" triggered unsolicited offers to create MC tasks. The fix is prompt-only: no code was changed in comms-responder.js (802 lines, unchanged) or telegram-agent.js.

Root cause (from audit.md): The original main-system-prompt.md placed the Task Actions section (lines 20-45) before conversational rules and had an over-broad MAYBE branch — "If unsure whether it is actionable, ASK" — which caused haiku-model responses to offer task creation for purely casual messages, since 20 live MC tasks were injected unconditionally into every context window.

Fix applied 2026-05-05:

MC #99331 added code-level gate (2026-05-05): CEO live test "Bok" still triggered "faza 2.5" hallucinations from MC task titles. While #99290 added intent classification to the PROMPT, comms-responder.js still injected mc.js list output unconditionally into every context window. #99331 added a CODE classifier that gates the injection: only task-create and status-query intents now receive the MC task list. This forms a two-layer defense (prompt + code) against context drift.

Key files:

2. Intent Categories

Every inbound message is classified into exactly one of these five categories before a response is generated. Classification is silent (not shown to the user).

CategoryTrigger signalsResponse modeTask action?
greetingBok, Selam, Zdravo, Hey, Ciao, opener phrasesShort friendly reply. No task mention.Never
chitchatKako si?, Šta radiš?, casual conversation1-2 sentences conversational. No task offer.Never
status-queryDa li je X gotov?, Šta ima novo?, Koji je status...?Pull from task context, answer directly and concisely.Never
questionKako radi X?, Zašto je Y?, factual or technical askAnswer from context. No task offer unless explicitly asked.Never
task-createKreiraj task, Napravi MC za..., Dodaj task, Otvori MC — explicit verb + task directiveCreate task, confirm with ID.ONLY this category

Default rule (prompt line 17): If category is not clearly task-create, respond conversationally. NEVER offer to create a task unprompted.

3. How It Works

The classification logic lives entirely in the system prompt file:
/Users/makinja/system/prompts/extracted/comms-responder/main-system-prompt.md

The prompt is loaded at runtime by comms-responder.js via the buildSystemPrompt() function. No intent classification code exists in the JS layer — the LLM (haiku to sonnet chain) performs STEP 1 classification silently before generating a response.

Request flow:

  1. CEO sends Telegram message.
  2. alai-telegram-agent.service (VM) receives update via Telegram Bot API webhook.
  3. telegram-agent.js calls comms-responder.js getResponse(message, history).
  4. comms-responder.js builds system prompt from main-system-prompt.md, injects scope block + task context + today's date.
  5. LLM classifies intent (STEP 1) silently, then generates response.
  6. If intent = task-create: response includes json:action block. telegram-agent.js parseAndExecuteActions() extracts and runs it against mc.js.
  7. For all other intents: plain conversational response, no action block.

Note on MC task injection: comms-responder.js lines 191-207 load 20 open MC tasks unconditionally into every context window (to enable status-query answers). This is by design; the classifier gate prevents this context from biasing non-status responses toward task creation.

3a. Code-Level Intent Gate (#99331)

While #99290 added prompt-based intent classification, CEO live testing on 2026-05-05 revealed that the greeting "Bok" still triggered hallucinations like "faza 2.5" from MC task titles injected into the context window. The root cause: comms-responder.js still executed mc.js list --limit 20 unconditionally for every message, regardless of intent.

MC #99331 added a CODE classifier (regex-based, 5 categories, lines 185-219) that runs before building the system prompt. The gate at line 234 injects MC task context ONLY if intent === 'task-create' || intent === 'status-query'. All other intents (greeting, chitchat, question) now receive zero MC task context in the prompt window.

Architecture: Two-Layer Defense

LayerLocationMethodPurpose
1. Code classifiercomms-responder.js, lines 185-219Regex pattern matching (5 categories)Gate MC task injection BEFORE prompt construction
2. Prompt classifiermain-system-prompt.md, STEP 1 tableLLM-driven (silent classification)Constrain response mode and action generation

The two layers work together as defense-in-depth: the CODE layer prevents irrelevant context from entering the window (token savings + drift prevention), while the PROMPT layer ensures the LLM's response adheres to the classified intent mode.

classifyIntent() Function

File: /Users/makinja/system/tools/comms-responder.js, lines 185-219
Input: Raw user message string (Bosnian or English)
Output: One of five strings: greeting, task-create, status-query, chitchat, question

Regex patterns (lines 190-209):

CategoryPatternExample matches
greeting/^(bok|selam|zdravo|hey|ciao)\b/iBok, Selam, Hey
task-create/(kreiraj|napravi|dodaj|otvori).*(task|mc)/iKreiraj task, Napravi MC za X
status-query/(da li|jel|je li).*(gotov|završen|done)/iDa li je X gotov?, Šta ima novo?
 /(šta|sta)\s+ima\s+novo/i (Unicode-aware, no \b)Šta ima novo?
chitchat/kako\s+(si|ste|ide|radi)/iKako si?, Kako ide?
question/(kako|zašto|šta|što|kada)/iKako radi X?, Zašto Y?

Note: The status-query pattern /(šta|sta)\s+ima\s+novo/i uses \s+ instead of \b (word boundary) because \b breaks on non-ASCII characters (š, č, ž). This was a critical fix during #99331 to support Bosnian phrases.

Gate Logic

File: /Users/makinja/system/tools/comms-responder.js, line 234
Code:

const intent = classifyIntent(options.userMessage);
let tasks = '';
if (intent === 'task-create' || intent === 'status-query') {
  const result = execSync(`node ${MC_PATH} list --limit 20`, { encoding: 'utf8' });
  tasks = result.trim();
}

Only task-create and status-query intents trigger the mc.js list exec. All other intents proceed with tasks = '' (empty string injected into prompt).

Behavioral Delta

ScenarioBEFORE #99331AFTER #99331Token savings
"Bok"20 MC tasks in prompt (~1000 chars)Zero MC tasks in prompt~250 tokens
"Kako si?"20 MC tasks in promptZero MC tasks in prompt~250 tokens
"Kako radi X?"20 MC tasks in promptZero MC tasks in prompt~250 tokens
"Šta ima novo?"20 MC tasks in prompt20 MC tasks in prompt0 (unchanged)
"Kreiraj MC za X"20 MC tasks in prompt20 MC tasks in prompt0 (unchanged)

Performance win: ~70% of CEO messages are greetings/chitchat/questions. Gate saves ~210ms latency (mc.js exec time) + ~250 tokens per message for these cases.

Why Two Layers?

  1. Defense-in-depth: Even if the LLM (prompt layer) drifts or hallucinates, the CODE layer has already filtered out irrelevant context from the window. The LLM cannot hallucinate "faza 2.5" if those task titles never entered the prompt.
  2. Performance: Skipping mc.js list for 70% of messages saves ~210ms per greeting/chitchat response.
  3. Token cost: Saves ~$0.0005/message for non-task intents (250 tokens × haiku rate).
  4. Correctness: Regex classification is deterministic — no LLM variance. The prompt layer adds nuance (e.g., handling ambiguous phrasing), but the CODE layer is the hard gate.

Rollback Artifact

VM backup: /opt/alai/system/tools/comms-responder.js.bak-99331-<timestamp>
Created at deploy time: 2026-05-05 22:47 UTC. This is the pre-gate version (unconditional mc.js list injection).

Rollback command:

az vm run-command invoke -g RG-ALAI-SUPPORT -n vm-alai-support \
  --command-id RunShellScript \
  --scripts "cp /opt/alai/system/tools/comms-responder.js.bak-99331-* /opt/alai/system/tools/comms-responder.js && systemctl restart alai-telegram-agent"

Warning: Rollback restores the unconditional MC task injection behavior — "Bok" will again trigger "faza 2.5" hallucinations. Only roll back if the CODE gate breaks status-query or task-create flows (regression evidence in /tmp/99331-evidence/regression-checklist.md).

4. Adding / Tuning Intent Patterns

Edit file: /Users/makinja/system/prompts/extracted/comms-responder/main-system-prompt.md

The STEP 1 classifier table (near the top of the prompt) is the authoritative classification surface.

  1. Open main-system-prompt.md on ANVIL.
  2. Locate the STEP 1 classifier table (lines 5-16 in current version).
  3. Add the new signal phrase to the Signals column of the relevant category row. Add a new row only if a genuinely new category is needed.
  4. If a new category should trigger a new action: also update the Task Actions section AND parseAndExecuteActions() in telegram-agent.js (requires CodeCraft dispatch — code change).
  5. Update the WHEN TO CREATE TASKS YES/NO examples block to include the new pattern.
  6. Run the test suite (Section 5) to validate no regression.
  7. Deploy per Section 6.

Critical constraint: Do NOT add a generic "if unsure, ASK" fallback. This was the direct cause of the original task-creation bias (audit.md, lines 42-44).

For CODE classifier tuning (classifyIntent regex patterns):

  1. Edit /Users/makinja/system/tools/comms-responder.js, lines 185-219.
  2. Add or modify regex pattern in the relevant if block.
  3. Unicode caution: use \s+ instead of \b for Bosnian patterns (š, č, ž break word boundaries).
  4. Test locally: node --check /Users/makinja/system/tools/comms-responder.js
  5. Deploy to VM per Section 6 (requires service restart — code change, not just prompt).
  6. Run live regression test (CEO "Bok" → no task mention).

5. Testing

Test suite: /tmp/99290-evidence/test-cases.js (10 message patterns)

Dry-run (no API cost — validates prompt content only):

node /tmp/99290-evidence/test-cases.js

Live API test (costs tokens — validates actual LLM classification):

node /tmp/99290-evidence/test-cases.js --live

10 test cases:

IDMessageExpected IntentTask Offer?Description
TC-01BokgreetingNoSingle greeting — must NOT trigger task creation offer
TC-02SelamgreetingNoBosnian greeting — must NOT trigger task creation offer
TC-03Šta ima novo?status-queryNoStatus query — should pull task context, NOT offer to create task
TC-04Kreiraj MC task za sintef follow-uptask-createYesExplicit task-create directive — MUST trigger create_task action
TC-05Kako si?chitchatNoCasual chitchat — must respond conversationally, no task offer
TC-06Da li je drop deploy gotov?status-queryNoDeploy status query — answer from context, no task offer
TC-07Napravi MC za UI bug na login stranicitask-createYesExplicit task-create (Bosnian variant) — MUST trigger create_task
TC-08Kako radi auth na Bilko-u?questionNoTechnical question — answer from context, no task creation
TC-09Ciao, šta radiš?greeting/chitchatNoMixed greeting+chitchat — conversational only
TC-10Otvori task za SINTEF LOI follow-up, prioritet Htask-createYesExplicit task-create with priority — MUST trigger create_task

7 prompt validation checks (dry run — all PASS after MC #99290 fix, per regression.md):

  1. Intent classifier table present (all 5 categories)
  2. Conversational default stated explicitly
  3. "ONLY for task-create intent" gate on Task Actions header
  4. Aggressive MAYBE/ASK removed (string absent from prompt)
  5. Greeting NO example present (Bok)
  6. Status-query NO example present (Šta ima novo)
  7. Task-create YES example present (Kreiraj task za sintef)

6. VM Deploy Procedure

Deploy target: Azure VM 4.223.110.181, service alai-telegram-agent.service
Last deploy: 2026-05-05 22:47 UTC
Code SHA-256: 233baff845b6c16153f900d1cab9756f84a72999f6425e375830a343b4870d36

  1. Edit /Users/makinja/system/prompts/extracted/comms-responder/main-system-prompt.md on ANVIL.
  2. Verify SHA-256 locally:
    shasum -a 256 /Users/makinja/system/prompts/extracted/comms-responder/main-system-prompt.md
  3. Copy updated prompt to VM:
    scp -i ~/.ssh/azure_alai /Users/makinja/system/prompts/extracted/comms-responder/main-system-prompt.md alai-admin@4.223.110.181:/home/alai-admin/system/prompts/extracted/comms-responder/main-system-prompt.md
  4. Restart daemon:
    ssh -i ~/.ssh/azure_alai alai-admin@4.223.110.181 "sudo systemctl restart alai-telegram-agent.service"
  5. Verify daemon active:
    ssh -i ~/.ssh/azure_alai alai-admin@4.223.110.181 "sudo systemctl status alai-telegram-agent.service --no-pager"
  6. Send test Telegram message (e.g., "Bok") and confirm bot replies with a greeting only — no task offer.
  7. Record new SHA-256 and timestamp in this runbook.

Scope: For prompt-only changes (intent tuning), only steps 1-7 are required. Code changes to comms-responder.js require a full service redeploy (not covered here).

For CODE changes (e.g., classifyIntent regex tuning):

  1. Edit /Users/makinja/system/tools/comms-responder.js on ANVIL.
  2. Verify syntax: node --check /Users/makinja/system/tools/comms-responder.js
  3. Compute SHA-256: shasum -a 256 /Users/makinja/system/tools/comms-responder.js
  4. Backup on VM:
    ssh -i ~/.ssh/azure_alai alai-admin@4.223.110.181 "sudo cp /opt/alai/system/tools/comms-responder.js /opt/alai/system/tools/comms-responder.js.bak-\$(date +%Y%m%d-%H%M%S)"
  5. Copy to VM:
    scp -i ~/.ssh/azure_alai /Users/makinja/system/tools/comms-responder.js alai-admin@4.223.110.181:/tmp/comms-responder.js
  6. Move to production:
    ssh -i ~/.ssh/azure_alai alai-admin@4.223.110.181 "sudo mv /tmp/comms-responder.js /opt/alai/system/tools/comms-responder.js"
  7. Restart daemon:
    ssh -i ~/.ssh/azure_alai alai-admin@4.223.110.181 "sudo systemctl restart alai-telegram-agent.service"
  8. Verify daemon active (step 5 above).
  9. Send live regression test (CEO "Bok" → no task mention).
  10. Record new SHA-256 and timestamp in this runbook.

7. Rollback

Backup artifact: /Users/makinja/system/prompts/extracted/comms-responder/main-system-prompt.md.bak-99290-20260505-213303
Created at deploy time: 2026-05-05 21:33:03 UTC. This is the pre-fix (biased) prompt.

  1. Locate backup on ANVIL:
    ls /Users/makinja/system/prompts/extracted/comms-responder/*.bak*
  2. Restore:
    cp /Users/makinja/system/prompts/extracted/comms-responder/main-system-prompt.md.bak-99290-20260505-213303 /Users/makinja/system/prompts/extracted/comms-responder/main-system-prompt.md
  3. Verify restored file SHA-256 differs from 33fc181...b5a2.
  4. Re-deploy to VM (Section 6, steps 3-6).
  5. Open a new MC task documenting the regression cause.

Warning: Rollback restores the task-creation bias. Only roll back if the new classifier causes explicit task-create messages (TC-04, TC-07, TC-10) to fail. Verify those three test cases pass before accepting rollback as stable.

8. Troubleshooting

SymptomLikely causeFix
Bot offers task creation for greetings or chitchatPrompt reverted or not deployed to VMVerify SHA-256 on VM matches 33fc181...b5a2. Re-deploy per Section 6.
Explicit "Kreiraj task" does NOT create a taskClassifier too tight or parseAndExecuteActions() failingRun TC-04 live test. Check journalctl -u alai-telegram-agent.service -n 50 on VM for parse errors.
Daemon not running on VMService crash, OOM, or failed deploysudo systemctl status alai-telegram-agent.service then sudo systemctl restart alai-telegram-agent.service
Bot not responding at allDaemon stopped, Telegram token invalid, or network issueCheck daemon status. Verify Telegram bot token in VM environment. Check VM network connectivity.
Wrong intent classified (e.g., question treated as status-query)Ambiguous phrasing at classifier boundaryAdd explicit example to the relevant NO/YES row in Section 2 of main-system-prompt.md. Re-test with test suite.
MC task created with wrong priorityNo priority pattern in prompt examplesCheck TC-10. Add "prioritet H/M/L" example to task-create YES examples in prompt.
"Bok" still triggers "faza 2.5" or task-title hallucinationCODE gate not deployed (#99331 fix missing)Verify comms-responder.js SHA-256 on VM = 233baff84.... Re-deploy code per Section 6 (code path).
Status-query ("Šta ima novo?") returns empty / "Nema taskova"CODE gate too tight or mc.js list exec failingCheck VM logs for mc.js errors. Verify gate condition (line 234) includes status-query.

VM log access:
ssh -i ~/.ssh/azure_alai alai-admin@4.223.110.181 "journalctl -u alai-telegram-agent.service -n 100 --no-pager"

Evidence files (MC #99290):

Evidence files (MC #99331):

SQLite DB Backup — Pillar #9 LITE

SQLite DB Backup — Pillar #9 LITE

Purpose: Restore drill procedure for SQLite databases backed up via azure-db-backup.sh wrapper extension (Build B of Pillar #9 LITE).

Related: Pillar #9 LITE spec, MC #99248

1. Wrapper Extension Overview

What was added: Build B extended ~/system/daemons/azure-db-backup.sh (existing 4h-interval LaunchAgent daemon) to include SQLite database snapshots alongside Docker volume backups.

Implementation (lines 466-471):

SQLITE_DBS=(
    "mission-control:$HOME/system/databases/mission-control.db"
    "hivemind:$HOME/system/databases/hivemind.db"
    "costs:$HOME/system/databases/costs.db"
    "knowledge:$HOME/system/databases/knowledge.db"
)

Process for each DB:

  1. sqlite3 .backup creates consistent snapshot (not file copy mid-write)
  2. gzip compression
  3. sha256 sidecar file generation
  4. Upload to Azure Blob Storage at sqlite/YYYY-MM-DD/<db-name>.db.gz
  5. Corresponding .sha256 sidecar uploaded

Storage location: Azure Storage Account alaibackups, container backups, blob prefix sqlite/<DATE>/

Evidence: First backup run on 2026-05-05 produced 8 blobs (4 databases + 4 sha256 sidecars):

2. Restore Drill Procedure

Use this procedure to validate backup integrity or perform disaster recovery.

Step 1: Download blob

az storage blob download \
  --account-name alaibackups \
  --container-name backups \
  --name "sqlite/2026-05-05/mission-control.db.gz" \
  --file /tmp/restore.db.gz \
  --auth-mode login

Step 2: Verify sha256 checksum

# Download sidecar
az storage blob download \
  --account-name alaibackups \
  --container-name backups \
  --name "sqlite/2026-05-05/mission-control.db.gz.sha256" \
  --file /tmp/restore.db.gz.sha256 \
  --auth-mode login

# Verify
sha256sum -c /tmp/restore.db.gz.sha256

Expected output: /tmp/restore.db.gz: OK

Step 3: Decompress

gunzip /tmp/restore.db.gz

Creates /tmp/restore.db

Step 4: Integrity check

sqlite3 /tmp/restore.db "PRAGMA integrity_check;"

Expected output: ok

Step 5: Sanity row count

# For mission-control.db
sqlite3 /tmp/restore.db "SELECT COUNT(*) FROM tasks;"

Expected: >10,000 tasks (baseline as of 2026-05-05: 10,797 rows)

# For hivemind.db
sqlite3 /tmp/restore.db "SELECT COUNT(*) FROM sessions;"

# For costs.db
sqlite3 /tmp/restore.db "SELECT COUNT(*) FROM runs;"

# For knowledge.db
sqlite3 /tmp/restore.db "SELECT COUNT(*) FROM entries;"

Step 6: Live restore (if DR scenario)

# Backup current DB first
cp ~/system/databases/mission-control.db ~/system/databases/mission-control.db.pre-restore-$(date +%s)

# Replace with restored copy
mv /tmp/restore.db ~/system/databases/mission-control.db

# Verify MC tool works
node ~/system/tools/mc.js list | head -5

3. Known Gap — RBAC Fix Needed

Issue: Service principal 1a0b3018 (used by azure-db-backup.sh) currently lacks the Microsoft.Compute/virtualMachines/runCommand/action permission on resource group alai-backups-rg.

Impact: Remote restore drills on Azure VM fail. Workaround: perform restore drill on ANVIL (local machine) after downloading blob.

Fix required (Azure admin or CEO):

# Get service principal object ID
az ad sp show --id 1a0b3018-xxxx-xxxx-xxxx-xxxxxxxxxxxx --query id -o tsv

# Assign Virtual Machine Contributor role at resource group scope
az role assignment create \
  --assignee <SP-OBJECT-ID> \
  --role "Virtual Machine Contributor" \
  --scope /subscriptions/<SUBSCRIPTION-ID>/resourceGroups/alai-backups-rg

Verification after fix:

az vm run-command invoke \
  --resource-group alai-backups-rg \
  --name vm-alai-support \
  --command-id RunShellScript \
  --scripts "sqlite3 --version"

Testing Schedule

TLDR Loop — Insight to Implementation Pipeline

TLDR Loop — Insight to Implementation Pipeline

Overview

The TLDR actionizer daemon closes the learning loop between daily TLDR email insights and ALAI's Mission Control task system. Instead of manually triaging insights or dumping them into an ever-growing backlog, the daemon automatically classifies, gates by relevance, routes to specialist owners, and creates tracked tasks — all without human intervention.

CEO Directive (2026-05-08): "Želim da ne sjedi u backlog i da se to krene u implementaciju" — No more backlog dumps. Every insight gets actionable routing or explicit discard.

Problem Solved

Daemon Flow Diagram

flowchart LR
    A[TLDR Email
09:00 daily] --> B[tldr-briefing.js
Extracts insights]
    B --> C[insights JSON log
~/system/logs/tldr-insights/YYYY-MM-DD.json]
    C --> D[tldr-actionizer.js
09:30 daily]
    D --> E{Ollama
TASK/SUGGEST/SKIP}
    E -->|SKIP| F[Discard
No MC task]
    E -->|SUGGEST| G2[Slack FYI only
No MC task]
    E -->|TASK| G{Ollama
HIGH/MED/LOW relevance}
    G -->|LOW| F
    G -->|HIGH or MED| H[OWNER_ROUTER
Keyword match]
    H --> I[MC Task Created
Priority M/L
TTL 30d]
    I --> J[Slack #exec
Summary]

OWNER_ROUTER Table

The daemon routes insights to specialist company owners based on keyword pattern matching. First match wins (order matters).

Pattern (regex, case-insensitive) Owner Example Keywords
security|breach|cve|vulnerab|ddos|exploit|malware|ransom|cyber securion CVE-2024-1234, DDoS attack, ransomware
llm|gpt|claude|llama|model|rag|agent|fine-tun|embed|inference|ollama agentforge GPT-5.5, RAG pipeline, Ollama fleet
payment|stripe|psd2|fintech|invoice|billing|bank|finance finverge Stripe API, PSD2 compliance, invoicing
docker|kubernetes|k8s|deploy|ci/cd|terraform|cloud run|aws|azure|gcp|nginx flowforge Cloud Run, Kubernetes, CI/CD pipelines
ios|android|mobile|swift|flutter|react native skybound Flutter 3.22, iOS performance, React Native
(No match) codecraft (default) Generic SaaS, backend, frontend

Safety Invariant: owner='backlog' is forbidden. The daemon will throw an error if any route resolves to 'backlog'. All tasks must go to a specialist owner or the default (codecraft).

Owner Assignment Rules

The daemon uses a two-stage Ollama LLM gate to determine owner and priority:

  1. Classification Gate (TASK / SUGGEST / SKIP)
    • SKIP: Generic advice not specific to ALAI's stack/products/market → Discard, no MC task
    • SUGGEST: Potentially valuable but needs CEO review before implementing → Slack summary only, no MC task created
    • TASK: Concrete, actionable, implementable within 1 week, clearly fits roadmap → MC task created
  2. Relevance Gate (HIGH / MED / LOW)
    • HIGH: Directly actionable for ALAI products, stack, or market focus (e.g., specific Ollama optimization, security patch for Node.js/Kotlin, feature for Bilko/Drop/Tok/Lobby)
    • MED: Broadly relevant to ALAI's industry or tech direction, warrants CEO awareness
    • LOW: Generic industry news with no clear ALAI connection → Discard, no MC task

Owner & Priority Decision Matrix

Relevance Classification Owner Priority MC Task?
HIGH TASK OWNER_ROUTER result M ✅ Yes (30d TTL)
HIGH SUGGEST (N/A) (N/A) ❌ No (Slack FYI only)
MED TASK OWNER_ROUTER result L ✅ Yes (30d TTL)
MED SUGGEST (N/A) (N/A) ❌ No (Slack FYI only)
LOW Any (Discarded) (No task) ❌ No

Key Rule: NEVER assign owner='backlog'. All tasks route to a specialist or CEO triage bucket.

Noise Prevention (2026-06-04)

MC #102890 | Problem: The daemon originally created an MC task for BOTH classification classes — TASK→"[TLDR] Implement" and SUGGEST→"[TLDR] Review". The SUGGEST/Review tasks accumulated unbounded. 63 noise tasks spanning 2026-04-23 through 2026-06-03 were triage-closed on 2026-06-03/04 (one-time manual cleanup).

The Fix: At-Source Prevention

SUGGEST class no longer creates MC tasks (since 2026-06-04). SUGGEST insights now appear only in the Slack summary as "Suggestions (FYI, no MC task)". Only TASK class creates an MC task, with a --ttl-minutes 43200 (30 days) backstop.

There is NO automatic age-based decay / bulk-close of existing tasks. The initial fix (2026-06-04) included a 14-day auto-decay mechanism that would bulk-close old [TLDR] tasks at daemon startup. CEO judged this too risky — it could silently close genuine "[TLDR] Implement" tasks that simply hadn't been picked up yet. The auto-decay logic was REMOVED per CEO decision.

Existing backlog is cleaned by manual/operator triage only, never silently. The 63 tasks closed on 2026-06-03/04 were a one-time manual cleanup, not an automated job.

The Casing Bug + Fix

Bug (historical): During initial implementation, the decay query used status='OPEN' (uppercase) but mission-control.db stores status lowercase ('open'). SQLite is case-sensitive for string comparisons, so the query matched 0 rows (silent no-op).

Fix: Normalized to status='open' (lowercase) and created_at threshold to 'YYYY-MM-DD HH:MM:SS' space-format. This verified the daemon's query mechanics were correct before the auto-decay feature was ultimately removed.

Operational Notes

Operator Runbook

How to Add New Domain Pattern

  1. Edit ~/system/daemons/tldr-actionizer.js and update the OWNER_ROUTER constant (lines 66-87).
  2. Add corresponding unit test in ~/system/daemons/tests/tldr-actionizer-router.test.js (mirror the pattern + test case).
  3. Run test: node ~/system/daemons/tests/tldr-actionizer-router.test.js — verify all tests pass.
  4. Restart daemon: launchctl stop com.john.tldr-actionizer && launchctl start com.john.tldr-actionizer

Example: To route design insights to vizu:

{
  pattern: /figma|sketch|design system|ui|ux|prototyp|wireframe/i,
  owner: 'vizu'
}

How to Override Routing for Specific Insight

If the daemon incorrectly routes an insight, manually reassign the MC task:

node ~/system/tools/mc.js assign <task_id> <new_owner>

How to Inspect Dry-Run Output

Test routing logic without creating real MC tasks:

node ~/system/daemons/tldr-actionizer.js --dry-run --date 2026-05-07

Output written to /tmp/tldr-router-dryrun-99824.json with classification, relevance, and assigned owner for each insight.

How to Verify Daemon Health

tail -50 ~/system/logs/tldr-actionizer.log | jq

Check for:

LaunchAgent: com.john.tldr-actionizer (runs daily at 09:30 Oslo time)

launchctl list | grep tldr-actionizer        # Check status
launchctl stop com.john.tldr-actionizer     # Manual stop
launchctl start com.john.tldr-actionizer    # Manual start

Backlog Sweep History

On 2026-05-08, MC #99823 performed a one-time sweep of existing backlog tasks to re-route or close them per the new routing rules. Results:

MC ID Priority Old Owner New Owner/Status Relevance Reasoning
9475MbacklogagentforgeHIGHPII redaction → Bilko/Drop/Lobby data pipelines
10081LbacklogclosedLOWHAL breach is news-only, no ALAI tie
99371LbacklogclosedLOWVideo gen — ALAI doesn't do video
99372LbacklogclosedLOWCodex cosmetic update — no action
99373LbacklogclosedLOWUK challenger bank — wrong market (we're Nordic/Balkan)
99374MbacklogsecurionMEDDDoS hardening — Securion service-line
99375MbacklogsecurionMEDTrellix breach awareness — Securion supply-chain
99565MbacklogagentforgeHIGHGPT-5.5 token cost — directly relevant to Pillar #9 cost ceiling
99566LbacklogclosedLOWVoice infra — ALAI doesn't do voice
99567MbacklogsecurionHIGHDeepsec — ALAI Sec service-line aligned (memo 2026-05-01)
99568LbacklogclosedLOWMeta-AI-research news — no concrete action

Summary: 11 tasks processed, 5 re-routed to specialist owners (agentforge, securion), 6 closed as low-relevance.

Known Gaps

1. ANVIL Ollama No Models Loaded

Impact: Daemon currently safe-fails ALL insights to alem/MED+SUGGEST because ANVIL Ollama (localhost:11434) has 0 models loaded. Classification and relevance gates return unclear responses, triggering default fallback.

Fix Options:

Status: Open (no fix deployed yet). Daemon runs daily but all insights currently default to CEO triage bucket.

2. OWNER_ROUTER Constant Drift Risk

Issue: The OWNER_ROUTER constant is defined in both:

If the daemon constant is updated without syncing the test file, unit tests become stale and may pass incorrectly.

Recommendation: Extract OWNER_ROUTER into a shared module:

// ~/system/daemons/lib/owner-router.js
module.exports = [ /* patterns */ ];

// In both files:
const OWNER_ROUTER = require('./lib/owner-router');

Status: Open (technical debt, does not block current operation).

Genesis MC IDs

Delivery Date: 2026-05-08 (initial), 2026-06-04 (noise prevention fix)

CEO Directive: "Želim da ne sjedi u backlog i da se to krene u implementaciju" — Implemented same-day.

Authored by Skillforge | ALAI Holding AS | Last updated 2026-06-04

IMAP → Paperless Archive Pipe (archive.alai.no)

IMAP → Paperless Archive Pipe (archive.alai.no)

Overview

This pipe automates archival of email attachments (contracts, invoices, signed documents) from ALAI's IMAP inboxes into the centralized Paperless-ngx document management system at archive.alai.no.

Use Cases:

Architecture

The pipeline consists of two independent CLI tools that can be chained:

┌──────────────────┐
│  email-inbox.db  │  (SQLite: all inboxes synced from one.com Dovecot IMAP)
└────────┬─────────┘
         │
         ▼
┌────────────────────────────────────────┐
│ email-attachment-fetcher.js            │  → /tmp/email-attachments/<msgid>/
│ (Extracts attachments from email DB)   │
└────────┬───────────────────────────────┘
         │
         ▼
┌────────────────────────────────────────┐
│ paperless-upload.js                    │  → HTTPS POST multipart/form-data
│ (Uploads file with metadata)           │
└────────┬───────────────────────────────┘
         │
         ▼  (3 headers: CF-Access-Client-Id, CF-Access-Client-Secret, Authorization)
         │
┌────────────────────────────────────────┐
│ archive.alai.no/api/documents/         │  (Paperless-ngx behind CF Access)
│ post_document/                          │
└─────────────────────────────────────────┘

Key Components:

Credentials

Item Name Bitwarden ID Purpose Fields
archive-alai-no CF Access e4fd63de-5989-4316-9092-1dfa72f2d2ee CF Access service token for archive.alai.no CF_ACCESS_CLIENT_ID, CF_ACCESS_CLIENT_SECRET
Paperless API Token — anvil 94227e4d-c55a-48fa-9421-05c649c5451e Paperless API authentication paperless_token

Fetching Credentials:

BW_SESSION=$(cat /tmp/bw-session)
CF_CLIENT_ID=$(bw get item e4fd63de-5989-4316-9092-1dfa72f2d2ee --session "$BW_SESSION" | jq -r '.fields[] | select(.name=="CF_ACCESS_CLIENT_ID") | .value')
CF_CLIENT_SECRET=$(bw get item e4fd63de-5989-4316-9092-1dfa72f2d2ee --session "$BW_SESSION" | jq -r '.fields[] | select(.name=="CF_ACCESS_CLIENT_SECRET") | .value')
PAPERLESS_TOKEN=$(bw get item 94227e4d-c55a-48fa-9421-05c649c5451e --session "$BW_SESSION" | jq -r '.fields[] | select(.name=="paperless_token") | .value')

Note: Both scripts auto-fetch credentials from Bitwarden when BW_SESSION environment variable is set or /tmp/bw-session exists.

Usage Examples

Example 1: Archive a Single Email's Attachment

Most common workflow — fetch attachment from email DB and upload to Paperless:

# Step 1: Find the email ID (search by subject or sender)
node ~/system/tools/email-inbox.js list --account alem --limit 20

# Step 2: Extract attachments (creates /tmp/email-attachments/<msgid>/)
node ~/system/tools/email-attachment-fetcher.js 5480

# Step 3: Upload to Paperless with metadata
node ~/system/tools/paperless-upload.js \
  --file "/tmp/email-attachments/<msgid>/SINTEF_LOI_signed.pdf" \
  --correspondent "SINTEF" \
  --document-type "Contract" \
  --tags "legal,signed,sintef" \
  --title "SINTEF Letter of Intent - Forskningsrådet Application"

Example 2: Archive Arbitrary File (Skip Email Fetch)

Upload any local file directly:

node ~/system/tools/paperless-upload.js \
  --file "/Users/makinja/Downloads/Invoice_12345.pdf" \
  --correspondent "SnowIT" \
  --document-type "Invoice" \
  --tags "billing,2026-05" \
  --title "SnowIT Monthly Invoice - May 2026"

Example 3: SINTEF LOI First-Run (Historical Reference)

Exact command used for first production run (2026-05-08):

# Email ID 5480 from alem@alai.no inbox
node ~/system/tools/email-attachment-fetcher.js 5480

# Extracted: /tmp/email-attachments/<9a646c02-c6c5-5f08-35fb-3ab4ec45d1c1@one.com>/SINTEF_LOI_signed.pdf

node ~/system/tools/paperless-upload.js \
  --file "/tmp/email-attachments/9a646c02-c6c5-5f08-35fb-3ab4ec45d1c1@one.com/SINTEF_LOI_signed.pdf" \
  --correspondent "SINTEF" \
  --document-type "Contract" \
  --tags "legal,signed,sintef,forskningsradet" \
  --title "SINTEF Letter of Intent - Forskningsrådet Application"

# Result: Paperless doc #127
# https://archive.alai.no/documents/127/

Example 4: Using Message-ID Instead of Email DB ID

node ~/system/tools/email-attachment-fetcher.js \
  --message-id "<9a646c02-c6c5-5f08-35fb-3ab4ec45d1c1@one.com>" \
  --account alem

Script Details

email-attachment-fetcher.js

Location: /Users/makinja/system/tools/email-attachment-fetcher.js
SHA-256: a3a03d83516c2cc44bb8b0a3753d5c41f0feb9aff54f93fef5a1bb9e3699d739

Syntax:

node email-attachment-fetcher.js <email_db_id>
node email-attachment-fetcher.js --message-id <mid> --account <account>

Output: /tmp/email-attachments/<msgid>/<filename1>, <filename2>, ...

paperless-upload.js

Location: /Users/makinja/system/tools/paperless-upload.js
SHA-256: d185ed2f3f7ec816cb68f2a421e5762219449ebda420653d1a2f16558d2e06dd

Syntax:

node paperless-upload.js --file <path> [OPTIONS]

Options:
  --correspondent NAME    Auto-creates if missing
  --document-type NAME    Auto-creates if missing
  --tags csv,list         Auto-creates if missing
  --title "Document Title"
  --no-poll               Skip task completion polling

Exit Codes:

Behavior:

CF Access Service-Token Rotation

Current Token:

Rotation Procedure:

  1. Log in to Cloudflare Dashboard → Zero Trust → Access → Service Auth
  2. Find policy for archive.alai.no
  3. Click "Create Service Token" → name it archive-pipe-YYYYMMv2
  4. Copy Client ID and Secret (shown only once)
  5. Update Bitwarden item e4fd63de-5989-4316-9092-1dfa72f2d2ee:
    • Replace CF_ACCESS_CLIENT_ID
    • Replace CF_ACCESS_CLIENT_SECRET
  6. Test with curl: 
    curl -I \
  -H "CF-Access-Client-Id: <new_id>" \
  -H "CF-Access-Client-Secret: <new_secret>" \
  "https://archive.alai.no/api/"
# Expected: HTTP 200 or 401 (not 302)
  7. If 200 → revoke old token in Cloudflare dashboard

Troubleshooting

HTTP 302 Redirect from archive.alai.no

Symptom: curl returns 302 Found to Cloudflare login page

Cause: Missing or expired CF Access service token

Fix:

  1. Verify token exists in Bitwarden item e4fd63de-5989-4316-9092-1dfa72f2d2ee
  2. Check token expiry in Cloudflare dashboard (Zero Trust → Service Auth)
  3. If expired → rotate per procedure above
  4. Verify script is passing headers (check paperless-upload.js code around line 40-60)

HTTP 401 Unauthorized from Paperless API

Symptom: paperless-upload.js exits with code 2

Cause: Invalid or missing Paperless API token

Fix:

  1. Verify token in Bitwarden item 94227e4d-c55a-48fa-9421-05c649c5451e
  2. Test token directly: 
    PAPERLESS_TOKEN="..."
curl -s -H "Authorization: Token $PAPERLESS_TOKEN" \
  -H "CF-Access-Client-Id: ..." \
  -H "CF-Access-Client-Secret: ..." \
  "https://archive.alai.no/api/correspondents/" | jq -r '.count'
  3. If null or error → regenerate token in Paperless UI (Settings → API Tokens) and update Bitwarden

Tag/Correspondent/Document-Type Creation Failures

Symptom: Script errors with "Failed to create correspondent X"

Cause: Paperless API permissions or schema validation failure

Fix:

  1. Check Paperless UI → ensure API user has documents.add_* permissions
  2. Verify tag/correspondent names don't contain invalid characters (use alphanumeric + spaces only)
  3. Check Paperless logs on Azure VM: 
    ssh -i ~/.ssh/azure_alai alai-admin@4.223.110.181
sudo docker logs paperless-webserver --tail 100

Email Attachment Not Found

Symptom: email-attachment-fetcher.js reports "No attachments found"

Causes:

Fix:

  1. Verify email exists: 
    node ~/system/tools/email-inbox.js show <id>
  2. Force IMAP sync: 
    node ~/system/tools/email-inbox.js sync --account alem
  3. Check attachment MIME parts in raw email (look for Content-Disposition: attachment)

File Upload Stalls (No Response After 30s)

Cause: Paperless task processing slow or stuck

Fix:

  1. Use --no-poll flag to skip task polling (upload completes instantly)
  2. Check document manually in Paperless UI after 1-2 minutes
  3. Restart Paperless workers if stuck: 
    ssh -i ~/.ssh/azure_alai alai-admin@4.223.110.181
sudo docker restart paperless-worker

Provenance

This runbook documents the IMAP→Paperless archive pipeline built and validated under:

Last Updated: 2026-05-08 | MC #100004 | Skillforge

LightRAG Stabilization Runbook — 2026-05-08

Genesis

On 2026-05-08 at 14:00, Kelsey Hightower reported LightRAG returning 502 errors. By 19:05 the service had degraded to complete timeout (000). Root cause: MainThread synchronous list comprehension in lightrag/lightrag.py:872 (apipeline_process_enqueue_documents) iterating over 125,341-record JsonDocStatusStorage on the asyncio event loop. A single POST to /documents/text triggered full file rewrite + pipeline iteration over 121K pending docs → CPU pegged at 100% → /health unreachable. The issue was compounded by running sbnb/lightrag:latest amd64-only image under Rosetta on Apple Silicon, incurring 2-3× performance tax.

Six-Step Fix Applied

S1: Disable Runaway Ingest Agents

Stopped LaunchAgents: com.alai.lightrag-outbox-ingest, com.alai.lightrag-migrate-pump, com.alai.lightrag-watchdog. Kept: keepwarm, backup, monitor.

S2: Prune Pending Queue

Stopped container, backed up doc_status.json. Filtered to status=processed only: 8,357 records retained; 116,986 pending/processing/failed quarantined to backup. Restarted container. CPU dropped to 0.31%.

S3: Verify Queryability

Tested naive mode + only_need_context=true (bypasses LLM, returns ALAI corpus chunks). Graph/label endpoint returned 200+ entities. Service functionally restored.

S4: Image Swap for Native ARM64

Replaced sbnb/lightrag:latest (amd64, v1.3.4) with ghcr.io/hkuds/lightrag:latest (native arm64, v1.4.16, official upstream). Verified via docker manifest inspect.

S5: Resource Limits

Added cgroup-enforced limits in compose: cpus: 2.0, memory: 4G.

S6: Re-Ingest Worker Design

Designed (not implemented) re-ingest worker with: batch_size=10, cooldown=60s, health-gate, pre-flight LLM availability check, cursor-based restart safety. Build gated on CEO OCD-3 (aging policy decision).

Verified Post-State

Known Follow-Ups

Evidence Files (Local, Transient)

References

Migadu Email Infrastructure — Add Domain & Alias Guide

Migadu Email Infrastructure — Alias & Mailbox Management

MC #100300 — 2026-05-10 | Owner: FlowForge (kelsey-hightower)

Replaces: CF Email Routing alias pattern. Migadu Mini ($90/yr) is now canonical for all ALAI email.

Account & API

Registered Domains (7)

alai.no | bilko.io | bilko.cloud | bilko.company | basicconsulting.no | basicfakta.no | getdrop.no

Active Mailboxes (5)

AddressBW Item NamePurpose
alem@alai.noMigadu — alem@alai.noCEO primary inbox
sales@bilko.ioMigadu — sales@bilko.ioBilko SR sales/lead
sales@bilko.cloudMigadu — sales@bilko.cloudBilko HR sales/lead
sales@bilko.companyMigadu — sales@bilko.companyBilko BA sales/lead
privacy@bilko.ioMigadu — privacy@bilko.ioBilko privacy requests

How to Add a New Alias

An alias delivers to an existing mailbox without creating a new inbox.

# 1. Get Migadu token from Bitwarden
BW_SESSION=$(cat /tmp/bw-session)
TOKEN=$(bw get password "78a41da0-b36f-46b9-b6e2-509b39768cec" --session "$BW_SESSION")

# 2. Create forwarding (alias) on an existing mailbox
# This adds contact@bilko.io -> delivered to sales@bilko.io mailbox
curl -X POST "https://api.migadu.com/v1/domains/bilko.io/mailboxes/sales/forwardings/" \
  -u "alem@alai.no:${TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"address":"contact@bilko.io","name":"Contact Alias"}'

# 3. Verify
curl -s "https://api.migadu.com/v1/domains/bilko.io/mailboxes/sales" \
  -u "alem@alai.no:${TOKEN}" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('forwardings', []))"

How to Add a New Mailbox

TOKEN=$(bw get password "78a41da0-b36f-46b9-b6e2-509b39768cec" --session "$(cat /tmp/bw-session)")

# Create new mailbox
curl -X POST "https://api.migadu.com/v1/domains/alai.no/mailboxes/" \
  -u "alem@alai.no:${TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"local_part":"hr","name":"HR Team","password":"","password_recovery_email":"alem@alai.no"}'

# Save password to Bitwarden immediately
echo '{"object":"item","type":1,"name":"Migadu — hr@alai.no","notes":"IMAP: imap.migadu.com:993 | SMTP: smtp.migadu.com:465","login":{"username":"hr@alai.no","password":"","uris":[]}}' | \
  bw encode | bw create item --session "$(cat /tmp/bw-session)"

How to Add a New Domain

TOKEN=$(bw get password "78a41da0-b36f-46b9-b6e2-509b39768cec" --session "$(cat /tmp/bw-session)")

# 1. Register domain
curl -X POST "https://api.migadu.com/v1/domains/" \
  -u "alem@alai.no:${TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"name":"newdomain.com"}'

# 2. Add DNS via CF (replace ZONE_ID):
CF_EMAIL="john@basicconsulting.no"
CF_KEY=$(bw get password "Cloudflare Global API Key" --session "$(cat /tmp/bw-session)")
ZONE_ID=""
DOMAIN="newdomain.com"

# MX records
curl -X POST "https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/dns_records" \
  -H "X-Auth-Email: ${CF_EMAIL}" -H "X-Auth-Key: ${CF_KEY}" -H "Content-Type: application/json" \
  -d "{\"type\":\"MX\",\"name\":\"${DOMAIN}\",\"content\":\"aspmx1.migadu.com\",\"priority\":10,\"ttl\":300}"
curl -X POST "https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/dns_records" \
  -H "X-Auth-Email: ${CF_EMAIL}" -H "X-Auth-Key: ${CF_KEY}" -H "Content-Type: application/json" \
  -d "{\"type\":\"MX\",\"name\":\"${DOMAIN}\",\"content\":\"aspmx2.migadu.com\",\"priority\":20,\"ttl\":300}"

# SPF
curl -X POST "https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/dns_records" \
  -H "X-Auth-Email: ${CF_EMAIL}" -H "X-Auth-Key: ${CF_KEY}" -H "Content-Type: application/json" \
  -d "{\"type\":\"TXT\",\"name\":\"${DOMAIN}\",\"content\":\"v=spf1 include:spf.migadu.com ~all\",\"ttl\":300}"

# DMARC
curl -X POST "https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/dns_records" \
  -H "X-Auth-Email: ${CF_EMAIL}" -H "X-Auth-Key: ${CF_KEY}" -H "Content-Type: application/json" \
  -d "{\"type\":\"TXT\",\"name\":\"_dmarc.${DOMAIN}\",\"content\":\"v=DMARC1; p=none; rua=mailto:postmaster@${DOMAIN}\",\"ttl\":300}"

# DKIM CNAMEs (x3)
for key in key1 key2 key3; do
  curl -X POST "https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/dns_records" \
    -H "X-Auth-Email: ${CF_EMAIL}" -H "X-Auth-Key: ${CF_KEY}" -H "Content-Type: application/json" \
    -d "{\"type\":\"CNAME\",\"name\":\"${key}._domainkey.${DOMAIN}\",\"content\":\"${key}.${DOMAIN}._domainkey.migadu.com\",\"ttl\":300}"
done

# 3. Poll until verified (typically 30-120 min)
watch -n 60 "curl -s https://api.migadu.com/v1/domains/${DOMAIN} -u alem@alai.no:\${TOKEN} | python3 -c \"import sys,json; d=json.load(sys.stdin); print(d.get('can_receive'))\""

Check Domain Verification

TOKEN=$(bw get password "78a41da0-b36f-46b9-b6e2-509b39768cec" --session "$(cat /tmp/bw-session)")

# All domains
curl -s "https://api.migadu.com/v1/domains/" -u "alem@alai.no:${TOKEN}" | \
  python3 -c "import sys,json; [print(f\"{d['name']}: state={d['state']}, can_receive={d['can_receive']}\") for d in json.load(sys.stdin)[\domains']]"

CF Email Routing (DISABLED)

CF Email Routing has been disabled on bilko.io, bilko.cloud, bilko.company (2026-05-10). Do NOT re-enable. MX is now Migadu. Old routing rules are inactive.

IMAP History Migration (imapsync)

Script: /Users/makinja/business/ALAI-Holding-AS/infrastructure/email-migadu-migration.sh

Run after all domains show can_receive=True. Source: imap.one.com:993 | 876 messages baseline (2026-05-10).

one.com Cancellation (CEO action)

  1. Confirm all Migadu domains active + 48h dual-host complete (2026-05-12T20:39Z)
  2. Remove one.com MX records from CF zones: alai.no (id: 2d2028ebbe8fe433390a894111f56016) + basicconsulting.no (ids: 6b5c01115411ed28165fe294141c17bc, e5f35554bf4976bc6d5a85d4a309ff05, ce32445b537e1fbfc9c1f7cc9f051092, 614c98d0c34862cb28a8b9eb48b29823)
  3. CEO logs into one.com -> My Products -> Cancel Email subscription

SnowIT.ba SaaS Funnel MVP

SnowIT.ba SaaS Funnel MVP — Production Runbook

1. Overview

What was delivered:

Why: CEO directive 2026-05-10 parallel to Bilko HR landing improvement session. snowit.ba was 108-line brochure with zero tracking, zero lead capture mechanism. MVP scope = establish funnel visibility (analytics) + lead capture (form + notification).

Context:

2. Architecture

Hosting & Deployment

Analytics Stack

Lead Form Pipeline

┌──────────────┐
│  Visitor     │
│  snowit.ba   │
└──────┬───────┘
       │ Fills form (name, email, message)
       │ index.html#contactForm
       ▼
┌─────────────────────────┐
│  api/contact.js         │
│  (Vercel Serverless)    │
│  • Honeypot validation  │
│  • Email regex check    │
│  • SMTP relay via       │
│    send.one.com         │
└──────┬──────────────────┘
       │ SMTP auth: info@basicconsulting.no
       │ To: info@snowit.ba
       ▼
┌───────────────────────┐
│  improvmx.com         │
│  MX forwarder         │
│  mx1/mx2.improvmx.com │
└──────┬────────────────┘
       │ Forward to
       ▼
┌──────────────────┐
│  enis@snowit.ba  │
│  (Lead recipient)│
└──────────────────┘

Parallel path (when Analytics enabled):
┌──────────────┐
│  Form submit │──▶ window.va('event', {name: 'Form Submit'})
└──────────────┘      │
                      ▼
              ┌────────────────────┐
              │ Vercel Analytics   │
              │ Dashboard ingestion│
              └────────────────────┘

Custom Events

Event Name Trigger Payload Location
Form Submit Contact form successful submission None (simple event) index.html line ~1773 (success callback)
CTA Click Click on .btn-primary, .btn-ghost, .nav-cta { label: string, href: string } index.html line ~1869 (global listener)

Code gating: Both event fire calls wrapped in if (window.va) check — events will queue and fire once Analytics enabled, no code change needed.

UTM Convention

Vercel Analytics auto-captures UTM params from URL query string. Recommended convention (documented in BUILD-BLUEPRINT.md):

Parameter Purpose Example Values
utm_source Channel linkedin, email, direct, referral, instagram, facebook
utm_medium Format social, email, organic, cpc, paid
utm_campaign Campaign identifier frizerski-landing-launch, bhtechlab-demo (kebab-case)
utm_content Variant/placement hero-a, footer-b, cta-variant-1

Example campaign URL:
https://snowit.ba/?utm_source=linkedin&utm_medium=social&utm_campaign=bhtechlab-demo&utm_content=hero-cta

3. Post-Deploy: Enable Vercel Web Analytics (MANUAL STEP — ONE-CLICK)

⚠️ CRITICAL MANUAL STEP REQUIRED

Analytics scripts return HTTP 404 until feature enabled in Vercel dashboard. This is a one-time, one-click operation (no payment required, FREE tier).

Step-by-Step Procedure

  1. Navigate to Vercel Analytics dashboard:
    URL: https://vercel.com/johns-projects-4b43bfa9/snowit-site/analytics
  2. Click "Enable Web Analytics" button
    Located in center of page. Button text may vary (e.g., "Enable Analytics" or "Get Started").
  3. Confirm FREE tier selection
    No payment method required for FREE tier. Limits: 2,500 events/month, 7-day data retention.
  4. Verify script now serves HTTP 200: 
    curl -sI https://snowit.ba/_vercel/insights/script.js | head -1
    Expected output: HTTP/2 200 (was HTTP/2 404 before enable)
  5. Generate test traffic:
    • Visit https://snowit.ba in browser (incognito/private mode to avoid cache)
    • Click hero CTA (triggers CTA Click event)
    • Fill and submit contact form (triggers Form Submit event)
  6. Confirm events appear in dashboard (5-10 min delay):
    Return to Analytics dashboard and verify:
    • Page view count incremented
    • Custom Events section shows "Form Submit" and "CTA Click" with count ≥ 1

Troubleshooting

Symptom Diagnosis Fix
Script still 404 after enable CDN propagation delay Wait 5 min, hard-refresh browser (Cmd+Shift+R / Ctrl+Shift+F5)
Events not appearing in dashboard Ingestion delay or ad blocker Wait 10 min. Test in incognito without extensions. Check browser console for errors.
"Enable Analytics" button missing Already enabled by another team member Check if dashboard shows "Analytics enabled" status. Verify script HTTP 200.

4. Operations

Dashboard Access

Lead Notification Flow

Auto-Reply to Submitter

Status: NOT IMPLEMENTED in v1 (MVP scope excluded this feature)

Lead submitter receives NO auto-reply confirmation email. Follow-on task opened to implement:

Note: Follow-on MC not yet created as of this runbook publication. Will be added when MC created.

5. Custom Events Reference

Adding New Custom Events

Custom events use Vercel Analytics window.va() API. Standard pattern:

if (window.va) {
  window.va('event', {
    name: 'Event Name Here'  // Required — string, max 50 chars
    // Optional properties (max 5 total):
    // label: 'button-text',
    // value: 42,
    // category: 'engagement'
  });
}

Current Events Implementation

Form Submit event (index.html line ~1773):

// Inside contactForm submit success callback:
if (window.va) {
  window.va('event', { name: 'Form Submit' });
}

CTA Click event (index.html line ~1869):

// Global event listener on DOMContentLoaded:
document.querySelectorAll('.btn-primary, .btn-ghost, .nav-cta').forEach(btn => {
  btn.addEventListener('click', function() {
    if (window.va) {
      const label = this.textContent.trim();
      const href = this.getAttribute('href') || this.getAttribute('data-href') || '';
      window.va('event', {
        name: 'CTA Click',
        label: label,
        href: href
      });
    }
  });
});

Viewing Events in Dashboard

  1. Navigate to Analytics dashboard
  2. Scroll to "Custom Events" section
  3. Events shown with count, trend graph (7-day retention on FREE tier)
  4. Click event name to see breakdown by label/value (if properties provided)

6. Verification Checklist

Production Health Check

1. Scripts deployed to all pages:

# Analytics script present in source
curl -s https://snowit.ba | grep -c "_vercel/insights"  # Expected: >= 1
curl -s https://snowit.ba/portfolio.html | grep -c "_vercel/insights"  # Expected: >= 1
curl -s https://snowit.ba/careers.html | grep -c "_vercel/insights"  # Expected: >= 1

2. CTAs consolidated (1 mailto per page, in footer only):

curl -s https://snowit.ba | grep -c "mailto:"  # Expected: 1
curl -s https://snowit.ba/portfolio.html | grep -c "mailto:"  # Expected: 1

3. Analytics enabled (after manual step in section 3):

curl -sI https://snowit.ba/_vercel/insights/script.js | head -1  # Expected: HTTP/2 200

4. Contact form functional:

5. Custom events firing (after Analytics enabled):

7. Known Gaps

Gap Impact Status MC ID
Auto-reply email to form submitter User receives no confirmation after submitting form (UX gap) Deferred to follow-on task TBD (not yet created)
Vercel Web Analytics dashboard ingestion Events fire in code but 404 on script load until manually enabled BLOCKED on manual one-click enable (section 3) MC #100302 (same task)
Vercel team access for enis@snowit.ba SnowIT CEO cannot view analytics dashboard without team invite Pending — requires manual Vercel invite from john@alai.no Not tracked (ops task, 2 min)
7-day data review Need data to decide if Vercel FREE tier sufficient or upgrade to Plausible (€9/mo) needed for richer attribution Scheduled revisit 2026-05-17 (7 days post-launch) TBD (calendar task, not MC)

8. Next Steps & Roadmap

Immediate (0-7 days)

  1. Enable Vercel Web Analytics (manual step, section 3) — BLOCKER
  2. Invite enis@snowit.ba to Vercel team as Viewer (2 min task)
  3. Monitor lead volume in enis@snowit.ba inbox (improvmx chain latency check)
  4. Collect 7 days of analytics data (page views, custom events, referrers, Web Vitals)

Week 2 (2026-05-17 onwards)

  1. Data review session with CEO Alem + Enis:
    • Analyze traffic sources (UTM attribution)
    • Form conversion rate (page views → form submits)
    • CTA Click patterns (which CTAs drive most engagement)
  2. Decision point: Keep Vercel FREE tier (2,500 events/mo, 7-day retention) OR upgrade to:
    • Vercel Pro ($20/mo) — unlimited events, 30-day retention, advanced filtering
    • Plausible.io (€9/mo) — GDPR-friendly, no cookie banner, full event stream export, unlimited retention
  3. Implement auto-reply email (if prioritized):
    • Template design (BS + EN locales)
    • Lexicon linguistic validation (Bosnian copy per ZAKON)
    • SMTP integration in api/contact.js

Deferred (post-funding or high lead volume)

9. References

Document Status: LIVE — Production Ready
Last Updated: 2026-05-10
Maintained By: Skillforge (ALAI Holding AS)
Contact: john@alai.no

Email fetcher: one.com → Migadu migration

Email Fetcher Migration: one.com → Migadu

MC Task: #100395
Migration Date: 2026-05-12 14:30-15:00 UTC
Verification: 12/12 atomic claims PASS (15:05 UTC)
Status: Production cutover complete

Overview

On 2026-05-10 20:20 UTC, the CEO registered Migadu email hosting and switched MX records for alai.no and basicconsulting.no to Migadu (priority 10/20), retaining one.com as fallback (priority 100). The email-agent daemon (~/system/tools/email-agent.js) and mail-native.js IMAP client were still configured for one.com IMAP, making mail routed to Migadu invisible to John's email processing pipeline. On 2026-05-12, John executed the migration cutover: provisioned 4 new Migadu mailboxes via Admin API, rotated Bitwarden credentials, reconfigured himalaya CLI and mail-native.js to use imap.migadu.com:993 + smtp.migadu.com:465, and deployed a workaround for himalaya 1.1.0 PLAIN SASL incompatibility by forcing ImapFlow fallback (HIMALAYA_DISABLED=1 env var).

Affected Components

Migadu Mailbox Provisioning

Migadu mailboxes are created via Admin API using the API key from Bitwarden item migadu keyy.

Create mailbox via API

# Get Migadu API credentials
API_KEY=$(bw get password 'migadu keyy' --session $(cat /tmp/bw-session))
DOMAIN="alai.no"  # or basicconsulting.no
LOCAL_PART="john"
PASSWORD=$(openssl rand -base64 24)

# Create mailbox
curl -X POST "https://api.migadu.com/v1/domains/${DOMAIN}/mailboxes" \
  -u "admin@alai.no:${API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "local_part": "'"${LOCAL_PART}"'",
    "password": "'"${PASSWORD}"'",
    "may_send": true,
    "may_receive": true,
    "may_access_imap": true,
    "may_access_pop3": true,
    "may_access_managesieve": true
  }'

# Verify creation
curl "https://api.migadu.com/v1/domains/${DOMAIN}/mailboxes/${LOCAL_PART}" \
  -u "admin@alai.no:${API_KEY}"

Store credentials in Bitwarden

# Create Bitwarden item with naming convention
echo "{
  \"organizationId\": null,
  \"folderId\": null,
  \"type\": 1,
  \"name\": \"Migadu — ${LOCAL_PART}@${DOMAIN}\",
  \"login\": {
    \"username\": \"${LOCAL_PART}@${DOMAIN}\",
    \"password\": \"${PASSWORD}\",
    \"uris\": [
      { \"match\": null, \"uri\": \"imap.migadu.com\" },
      { \"match\": null, \"uri\": \"smtp.migadu.com\" }
    ]
  }
}" | bw encode | bw create item --session $(cat /tmp/bw-session)

Evidence: See /Users/makinja/system/state/evidence/migadu-mailbox-create-20260512T083653Z.log for actual execution output of 4 mailboxes created on 2026-05-12.

Bitwarden Credential Structure

Naming convention: Migadu — <email address>

This convention is hardcoded in ~/system/tools/mail-native.js VAULT_NAMES mapping:

const VAULT_NAMES = {
  john: 'Migadu — john@basicconsulting.no',
  info: 'Migadu — info@basicconsulting.no',
  alai: 'Migadu — john@alai.no',
  alem: 'Migadu — alem@alai.no',
  dev: 'Migadu — dev@alai.no',
  gmail: 'Gmail — alembasic@gmail.com'
};

List all Migadu credentials:

bw list items --search 'Migadu —' --session $(cat /tmp/bw-session) | jq -r '.[] | "\(.name) (\(.id))"'

End-to-End Verification

Use Python imaplib to verify IMAP login and mailbox access:

#!/usr/bin/env python3
import imaplib
import json
import subprocess

def test_imap(email):
    # Fetch password from Bitwarden
    vault_name = f"Migadu — {email}"
    bw_session = open('/tmp/bw-session').read().strip()
    password = subprocess.check_output(
        ['bw', 'get', 'password', vault_name, '--session', bw_session],
        text=True
    ).strip()
    
    # Connect to Migadu IMAP
    imap = imaplib.IMAP4_SSL('imap.migadu.com', 993)
    imap.login(email, password)
    status, messages = imap.select('INBOX')
    
    if status == 'OK':
        msg_count = int(messages[0])
        print(f"✓ {email}: INBOX OK, {msg_count} messages")
    else:
        print(f"✗ {email}: SELECT INBOX failed")
    
    imap.logout()

if __name__ == '__main__':
    accounts = [
        'alem@alai.no',
        'john@alai.no',
        'john@basicconsulting.no',
        'info@basicconsulting.no',
        'dev@alai.no'
    ]
    for acc in accounts:
        test_imap(acc)

Expected output:

✓ alem@alai.no: INBOX OK, 881 messages
✓ john@alai.no: INBOX OK, 0 messages
✓ john@basicconsulting.no: INBOX OK, 13 messages
✓ info@basicconsulting.no: INBOX OK, 0 messages
✓ dev@alai.no: INBOX OK, 0 messages

Evidence: Verifier executed equivalent IMAP4_SSL login probes on 2026-05-12 15:05 UTC — claim C3 PASS (see /Users/makinja/system/state/evidence/mc-100395-verifier-verdict-20260512.md).

Rollback Procedure

If Migadu migration must be reverted (e.g., service outage, credential issues), follow these steps:

  1. Stop email-agent daemon: 
    launchctl unload ~/Library/LaunchAgents/com.john.email-agent.plist
  2. Restore himalaya config to one.com: 
    cp ~/.config/himalaya/config.toml ~/.config/himalaya/config.toml.migadu-backup-$(date +%Y%m%d-%H%M%S)
cp ~/.config/himalaya/config.toml.one-com-backup-20260512-163802 ~/.config/himalaya/config.toml
  3. Revert mail-native.js (verify before running): 
    cd ~/system/tools
git diff mail-native.js  # Review changes
git checkout HEAD -- mail-native.js  # Revert to pre-migration state
  4. Remove HIMALAYA_DISABLED from LaunchAgent: 
    cp ~/Library/LaunchAgents/com.john.email-agent.plist ~/Library/LaunchAgents/com.john.email-agent.plist.bak-$(date +%Y%m%d-%H%M%S)
# Edit plist to remove HIMALAYA_DISABLED env var:
plutil -replace EnvironmentVariables.HIMALAYA_DISABLED -string "" ~/Library/LaunchAgents/com.john.email-agent.plist
# Or manually edit and remove the key/value pair
  5. Update Bitwarden vault names in code (if needed):

    If mail-native.js git revert doesn't restore old BW item names, manually edit VAULT_NAMES back to Email - <addr> pattern.

  6. Restart daemon: 
    launchctl load ~/Library/LaunchAgents/com.john.email-agent.plist
  7. Verify connection to one.com: 
    tail -f ~/system/logs/email-agent.log | grep -E "Connected|ERROR"
    Expect lines like: Connected to john (john@basicconsulting.no) within 60 seconds.

Rollback time estimate: 5-10 minutes (assuming backups are intact).

Known Issue: himalaya 1.1.0 PLAIN SASL vs Migadu

Symptom: himalaya CLI 1.1.0 fails IMAP login to imap.migadu.com:993 with error:

Error: cannot parse envelope at line 1 near column 1
Kind: MalformedMessage

Root cause: himalaya 1.1.0 attempts PLAIN SASL authentication, which Migadu's IMAP server rejects or mishandles (verify before re-running). This is a known incompatibility between himalaya's IMAP library and Migadu's Dovecot configuration.

Workaround: Force email-agent.js to skip himalaya and use ImapFlow (native Node.js IMAP client) by setting environment variable:

<key>EnvironmentVariables</key>
<dict>
  <key>HIMALAYA_DISABLED</key>
  <string>1</string>
</dict>

in ~/Library/LaunchAgents/com.john.email-agent.plist.

Evidence: Email-agent.js log on 2026-05-12 14:48:12 shows:

{"timestamp":"2026-05-12T14:48:12.896Z","service":"email-agent","level":"info","message":"[WARN] himalaya disabled for john, falling back to legacy unseen fetch"}

All 6 mailboxes connected successfully via ImapFlow (claims C9/C10 PASS).

Long-term fix: File upstream issue with himalaya maintainers or test downgrade to himalaya 1.0.x. Track in separate MC task.

Migration Timeline

Timestamp (UTC)Event
2026-05-10 20:20CEO registered Migadu, MX records switched
2026-05-12 08:364 mailboxes provisioned via Migadu API (post, dev, info, john@basicconsulting)
2026-05-12 14:37john@alai.no mailbox created
2026-05-12 14:48email-agent cutover: himalaya disabled, ImapFlow connected to 5 Migadu + 1 Gmail
2026-05-12 14:51First mail cycle after cutover: 3 new emails classified via Ollama
2026-05-12 15:05Verifier subagent: 12/12 claims PASS

Post-Migration State

References

Paperless-ngx — CF Access SSO Setup Plan

Implementation

Now available as skill /cf-access-sso. Execute via Skill tool with args: subdomain, service, container, vm_rg, vm_name, cf_user_email, [service_token_id]. Manual paste-ready commands below remain as fallback.

Skill path: ~/.claude/skills/cf-access-sso/SKILL.md

Invoke example:

Skill('cf-access-sso',
  subdomain='archive.alai.no',
  service='paperless',
  container='alai-paperless-1',
  vm_rg='RG-ALAI-SUPPORT',
  vm_name='vm-alai-support',
  cf_user_email='alembasic@gmail.com',
  service_token_id='9d63505b-2e07-49e4-beb6-28b545a93bef'
)

Skill handles: pre-flight checks, user rename, env apply, container restart, CF Access app creation (with service token bypass + email allow policies), verification gate (curl 302 + Playwright screenshot), rollback script emission. Evidence written to: /tmp/evidence-cf-sso-paperless/

Paperless-ngx — CF Access SSO Setup Plan

STATUS: PLAN — NOT YET EXECUTED Written: 2026-05-15 by John (AI Director) Execution: CEO terminal (az vm run-command + CF API) Prerequisite: review this page fully before executing

Current State (verified 2026-05-15)

Component Current Config
URL https://archive.alai.no
CF Access app "All ALAI Services" wildcard *.alai.no (id: cd7cf0f0)
Dedicated archive app None — wildcard catches all
IdP Email OTP only (alai-no.cloudflareaccess.com)
Human login Username + password (user alembasic, superuser)
API auth DRF Token (c9ec30192db3c95802349335edea4bca864a937a)
IMAP pipe auth CF service token (BW: e4fd63de) + Paperless API token
SSO Not configured
Browser access IP bypass fires for LAN (92.221.168.61) — no CF auth challenge

Key finding: CF Access only injects Cf-Access-Authenticated-User-Email when the allow policy fires. When IP bypass matches first, no identity header is set. Current bypass-first config means SSO cannot work for LAN browser sessions without restructuring the CF app.

Architecture Decision: Dedicated CF Access App

Create a separate CF Access app for archive.alai.no that:

The wildcard *.alai.no app continues to handle all other services and IP-bypass API access.

Header Chain (after SSO enabled)

CEO Browser
    ↓
Cloudflare CF Access (Email OTP challenge — once per 24h)
    ↓  injects: Cf-Access-Authenticated-User-Email: alembasic@gmail.com
Caddy reverse proxy (archive.alai.no → paperless:8000)
    ↓  forwards all headers by default
Paperless-ngx (Django) reads: HTTP_CF_ACCESS_AUTHENTICATED_USER_EMAIL
    ↓  matches username "alembasic@gmail.com" → auto-login
CEO is logged in, no password prompt

Execution Script

Run from CEO terminal (has full az auth). Do NOT execute all at once — verify each phase.

Phase 1: Rename Paperless user (preserve document ownership)

# SSH to Azure VM
ssh -i ~/.ssh/azure_alai alai-admin@4.223.110.181

# Rename 'alembasic' → 'alembasic@gmail.com'
docker exec alai-paperless-1 python manage.py shell -c "
from django.contrib.auth import get_user_model
User = get_user_model()
u = User.objects.get(username='alembasic')
print('Before:', u.username, u.email)
u.username = 'alembasic@gmail.com'
u.email = 'alembasic@gmail.com'
u.save()
print('After:', u.username)
"
# Expected output: After: alembasic@gmail.com

# Verify: list users
docker exec alai-paperless-1 python manage.py shell -c "
from django.contrib.auth import get_user_model
for u in get_user_model().objects.all():
    print(u.id, u.username, u.is_superuser, u.is_active)
"

Phase 2: Update Paperless env vars for trusted-header SSO

# On Azure VM — find docker compose file
ls /opt/alai/ /home/alai-admin/ 2>/dev/null
# Likely: /opt/alai/docker-compose.yml or /home/alai-admin/docker-compose.yml

# Add/update these env vars in the paperless service:
# PAPERLESS_ENABLE_HTTP_REMOTE_USER=true
# PAPERLESS_HTTP_REMOTE_USER_HEADER_NAME=HTTP_CF_ACCESS_AUTHENTICATED_USER_EMAIL

# Example edit (adjust path as needed):
# In docker-compose.yml, under paperless service environment:
#   - PAPERLESS_ENABLE_HTTP_REMOTE_USER=true
#   - PAPERLESS_HTTP_REMOTE_USER_HEADER_NAME=HTTP_CF_ACCESS_AUTHENTICATED_USER_EMAIL

# Restart paperless (NOT the whole stack — don't restart redis/gotenberg/tika):
docker compose -f /path/to/docker-compose.yml restart alai-paperless-1

Phase 3: Verify Caddy forwards the header

# Test from Azure VM (loopback):
# Simulate what CF Access would inject:
curl -s -o /dev/null -w "%{http_code}" \
  -H "Cf-Access-Authenticated-User-Email: alembasic@gmail.com" \
  -H "Cf-Access-JWT-Assertion: test" \
  http://localhost:8000/accounts/login/
# This should NOT auto-login (no Caddy = no trusted proxy check) — that's expected
# The real test is through Caddy (HTTPS from browser)

# Check Caddy config:
cat /opt/alai/Caddyfile 2>/dev/null || docker exec alai-caddy-1 cat /etc/caddy/Caddyfile 2>/dev/null
# Verify archive.alai.no block does NOT strip headers explicitly
# Caddy default: all request headers are forwarded to upstream

Phase 4: Create dedicated CF Access app for archive.alai.no

# Use CF API to create the dedicated app
CF_ACCOUNT_ID="d0ac2afb6bb5b298723b85a114151a04"
CF_EMAIL="john@basicconsulting.no"
CF_API_KEY="$(bw get item 'Cloudflare Global API Key' --session $(cat /tmp/bw-session) | jq -r '.login.password')"
OTP_IDP_ID="ff0a28e6-2220-4de2-a82f-48385d88b163"
PIPE_TOKEN_ID="9d63505b-2e07-49e4-beb6-28b545a93bef"

curl -s -X POST \
  "https://api.cloudflare.com/client/v4/accounts/$CF_ACCOUNT_ID/access/apps" \
  -H "X-Auth-Email: $CF_EMAIL" \
  -H "X-Auth-Key: $CF_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "archive.alai.no — Paperless SSO",
    "domain": "archive.alai.no",
    "type": "self_hosted",
    "session_duration": "24h",
    "auto_redirect_to_identity": false,
    "http_only_cookie_attribute": true,
    "same_site_cookie_attribute": "lax",
    "app_launcher_visible": true,
    "allowed_idps": ["'"$OTP_IDP_ID"'"],
    "policies": [
      {
        "name": "archive-pipe service token bypass",
        "decision": "bypass",
        "precedence": 1,
        "include": [{"service_token": {"token_id": "'"$PIPE_TOKEN_ID"'"}}]
      },
      {
        "name": "CEO alembasic access",
        "decision": "allow",
        "precedence": 2,
        "include": [{"email": {"email": "alembasic@gmail.com"}}]
      }
    ]
  }'
# Save the returned app id — needed if you want to update or delete this app

Phase 5: Verify SSO works

# From CEO browser (Mac Air, NOT Mac Studio with VPN):
# 1. Clear cookies for archive.alai.no
# 2. Navigate to https://archive.alai.no
# 3. Should see CF Access OTP challenge — enter alembasic@gmail.com
# 4. Enter OTP from email
# 5. Should land directly on Paperless dashboard (logged in as alembasic@gmail.com)
# 6. Check: Profile → Settings — should show alembasic@gmail.com as username

# API/pipe verification (no regression):
source ~/.config/alai/paperless-token.env
curl -s --interface "$PAPERLESS_BIND_INTERFACE" \
  -H "Authorization: Token $PAPERLESS_TOKEN" \
  "$PAPERLESS_BASE/api/documents/?page_size=1" | grep '"count"'
# Should return document count — confirms API token auth still works

Rollback Procedure

If SSO breaks login:

# Method 1: Disable SSO via env (SSH or az run-command)
# Edit docker-compose.yml: set PAPERLESS_ENABLE_HTTP_REMOTE_USER=false
# docker compose restart alai-paperless-1
# Then login with alembasic@gmail.com + password

# Method 2: Emergency password reset (if locked out completely)
az vm run-command invoke \
  --resource-group RG-ALAI-SUPPORT \
  --name vm-alai-support \
  --command-id RunShellScript \
  --scripts "docker exec alai-paperless-1 python manage.py changepassword alembasic@gmail.com"

# Method 3: Delete the dedicated CF Access app (reverts to wildcard + IP bypass)
# Get the app id from Phase 4 output, then:
curl -s -X DELETE \
  "https://api.cloudflare.com/client/v4/accounts/$CF_ACCOUNT_ID/access/apps/<APP_ID>" \
  -H "X-Auth-Email: $CF_EMAIL" \
  -H "X-Auth-Key: $CF_API_KEY"

Risk Table

Risk Likelihood Mitigation
API token breaks after user rename Low Tokens bound to DB user ID (int), not username
Caddy strips CF header Low Default Caddy forwards all headers; verify Caddyfile
CEO locked out after SSO enable Medium Emergency: az run-command changepassword
IMAP pipe breaks Low Pipe uses service token + API token, unaffected by SSO
OTP fatigue Low 24h session — one OTP per day max
*.alai.no wildcard still matches Low Exact-match app takes CF routing precedence
SSO header spoofing Low CF Access validates JWT; only CF can inject this header. Caddy only listens on localhost

What We Are NOT Doing

Appendix: Auth Strategy — Internal Phase (current)

Updated: 2026-05-16 by John (AI Director)

Status: ACTIVE — Email OTP only, 30-day sessions. Google OAuth deferred.

Auth Strategy (Internal Phase — current)

Session duration change evidence:

Root Cause of Email OTP Failure (resolved 2026-05-16)

CF Access evaluates the allow policy before dispatching the OTP email. The original policy only had alembasic@gmail.com. When CEO entered alem@alai.no, CF rejected the request at the policy gate — no email was ever dispatched to Migadu. Migadu mailbox was healthy throughout.

Fix: policy updated to include all 3 CEO aliases. Policy ID: a9e36b92-5158-4ced-a333-a8d84a67a705.

Client-facing IdP Strategy — Deferred

Google OAuth IdP setup is deferred until the client-facing phase. Manual Google Cloud Console setup is not justified for 1 internal user when Email OTP + 30-day sessions already provides low-friction access.

Trigger to upgrade IdP:

When triggered — build path:

Until then: Email OTP scales to approximately 10 internal users without UX regression, given 30-day session duration.

IdP Tiers (target state — not yet active)

Tier Who Primary IdP Fallback Status
ALAI Staff CEO + internal team Email OTP (30d session) — ACTIVE
SME Clients SnowIT and similar Email OTP — Future
Enterprise Clients Custom per-client SAML 2.0 / OIDC Email OTP Future

OCD-Delta Webhook Runbook — Anti-Hallucination V2

OCD-Delta Webhook Runbook — Anti-Hallucination V2

Component: OCD-Delta (Orchestrator Claim Detector — Delta)
Source spec: Anti-Hallucination V2 §4 (Secondary Hardening)
MC: #99732
Published: 2026-05-22

Overview

The OCD-Delta webhook fires on Task tool PostToolUse. It detects verdict claims in agent text output and blocks propagation if the verdict does not satisfy the V2 contract. This closes the gap where Proveo text claims PASS but the orchestrator accepts the claim before any gate fires (MC #99595 failure mode).

Trigger

Blocking Conditions

OCD-Delta blocks (exits non-zero) when ANY of:

Workaround for PostToolUse Limitation

Claude Code does not expose raw Task response text to bash hooks directly. Protocol:

  1. Agent writes verdict JSON to /tmp/ocd-delta-manifest-<mc_id>.json before returning its response
  2. OCD-Delta reads this manifest file
  3. Hook validates and exits 0 (allow) or 1 (block)
  4. If manifest absent: hook logs warning and allows (backward-compatible)

Verdict TTL Check

EXPIRES_AT=$(jq -r .expires_at /tmp/ocd-delta-manifest-<mc_id>.json)
NOW=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
if [[ "$NOW" > "$EXPIRES_AT" ]]; then
  echo "ERROR: Verdict expired at $EXPIRES_AT. NULL verdict — rerun required."
  exit 1
fi

Machine Check Count Validation

COUNT=$(jq .machine_check_count /tmp/ocd-delta-manifest-<mc_id>.json)
EXECUTED=$(jq .machine_checks_executed /tmp/ocd-delta-manifest-<mc_id>.json)
if [[ "$EXECUTED" -lt "$COUNT" ]]; then
  echo "ERROR: machine_checks_executed ($EXECUTED) < machine_check_count ($COUNT). Verdict invalid."
  exit 1
fi

GO-LIVE-READY Quorum Check

VERDICT=$(jq -r .verdict /tmp/ocd-delta-manifest-<mc_id>.json)
if [[ "$VERDICT" == "GO-LIVE-READY" ]]; then
  MATCHES=$(jq -r .john_reproducer_output.matches_verdict /tmp/ocd-delta-manifest-<mc_id>.json)
  if [[ "$MATCHES" != "true" ]]; then
    echo "ERROR: GO-LIVE BLOCKED — john_reproducer_output.matches_verdict is not true. ZAKON #29.2 violation."
    exit 1
  fi
fi

Installation

  1. Script: ~/.claude/hooks/ocd-delta-validator.sh
  2. Register in Claude Code settings as PostToolUse hook scoped to Task tool
  3. Make executable: chmod +x ~/.claude/hooks/ocd-delta-validator.sh
  4. Test: create /tmp/ocd-delta-test.json with missing evidence_files, run hook, expect exit 1

Monthly Hallucination Drill

LaunchAgent: com.alai.hallucination-drill
Plist: ~/Library/LaunchAgents/com.alai.hallucination-drill.plist
Schedule: monthly

Drill sequence: generate synthetic verdict with PASS claim but false intent_proof → run through OCD-Delta → expect HALLUCINATION_DETECTED (exit 1). If hook exits 0: auto-create P0 MC, block all H/BLOCKER task closes until patched.

Escalation

When blocked: print error to stderr with MC ID and reason. If verdict=REFUSED: auto-post to Slack #john-alerts within 15 minutes. Suspend all dependent task completions until CEO arbitrates.

Source: Anti-Hallucination V2 §4 | MC #99732 | Cross-ref: BookStack page 2995 (full spec)

Deterministic Session Summary Compiler Runbook

Deterministic Session Summary Compiler Runbook

MC: #101065
Status: partial implementation, pending independent validation and final closure.

Purpose

Session-end checkpoints must be generated from machine evidence, not John-authored narrative. The compiler creates a checkpoint from git, Mission Control, evidence ledger, cost tracker, claim schemas, and transcript timestamps. John may only add one constrained notes paragraph.

Runtime pieces

Generator inputs

By default the generator reads:

Test overrides are available for deterministic fixtures:

Manual commands

Syntax checks:

node --check /Users/makinja/system/tools/session-summary-generator.js
bash -n /Users/makinja/system/hooks/session-summary-stop-hook.sh
node --check /Users/makinja/system/tests/session-summary-generator-simulation.js

Manual dry-run:

node /Users/makinja/system/tools/session-summary-generator.js \
  --session-start 2026-05-23T13:45:00Z \
  --session-id manual-smoke-101065 \
  --auto-write \
  --output /tmp/alai/session-summary-generator-smoke-101065.md

Seeded simulation:

node /Users/makinja/system/tests/session-summary-generator-simulation.js

Expected seeded simulation evidence:

Stop hook behavior

session-summary-stop-hook.sh reads hook JSON on stdin, invokes the generator with --auto-write, logs errors, and always exits 0. Session closing must not be blocked by checkpoint generation failure.

If checkpoint generation fails, inspect:

tail -50 /Users/makinja/.cache/alai/session-summary-errors.log

John Notes rule

John Notes are optional and constrained by the generator:

The generated machine sections above ## John Notes are not to be hand-edited.

Closure requirements

Do not mark MC #101065 done until:

  1. independent Proveo validation passes or accepted caveats are documented;
  2. Stop hook registration is confirmed live;
  3. seeded simulation output is retained as evidence;
  4. BookStack/runbook publication is confirmed;
  5. any remaining liveness-validator integration decision is recorded.

Cross-company Workflow Runner v0

Cross-company workflow runner v0

Last verified: 2026-05-25

What it is

~/system/tools/cross-company-workflow.js is a deterministic wrapper around Mission Control and Company Mesh.

Use it when a task needs a bounded multi-company loop such as:

It is not a replacement for Mission Control ownership, Mehanik/Prompt-Forge gates, or Proveo verification. It only makes the cross-company handoff repeatable and evidence-producing.

Source files

Commands

Validate a workflow:

node ~/system/tools/cross-company-workflow.js validate ~/system/workflows/company-mesh-tool-smoke.json --json

Run a workflow:

node ~/system/tools/cross-company-workflow.js run ~/system/workflows/company-mesh-tool-smoke.json --json

Inspect state:

node ~/system/tools/cross-company-workflow.js status /tmp/alai/cross-company-workflows/<state>.json --json

Finalize MC tasks after a PASS workflow:

node ~/system/tools/cross-company-workflow.js finalize /tmp/alai/cross-company-workflows/<state>.json \
  --bookstack https://docs.alai.no/link/184 \
  --json

For static plumbing-only responder runs, finalization is intentionally blocked unless explicitly marked:

node ~/system/tools/cross-company-workflow.js finalize /tmp/alai/cross-company-workflows/<state>.json \
  --bookstack https://docs.alai.no/link/184 \
  --allow-plumbing-finalize \
  --json

Workflow JSON shape

Minimum example:

{
  "name": "example-review-loop",
  "description": "Bounded build/security/QA review loop.",
  "priority": "M",
  "actor": "john",
  "responderMode": "gemini-review",
  "steps": [
    {
      "id": "codecraft",
      "company": "CodeCraft",
      "title": "CodeCraft review",
      "purpose": "Review implementation approach.",
      "prompt": "Review the pasted evidence. Return PASS/PARTIAL/BLOCKED first.",
      "endState": "ANSWERED"
    },
    {
      "id": "securion",
      "company": "Securion",
      "title": "Security review",
      "purpose": "Review security/privacy risks.",
      "dependsOn": ["codecraft"],
      "prompt": "Review security implications of the pasted evidence.",
      "endState": "ANSWERED"
    }
  ]
}

Required per step:

Optional per step:

Safety defaults

The runner is intentionally conservative:

Responder modes

Mode Meaning Use
answer Static deterministic response Plumbing-only smoke tests
blocked Static blocked response Negative-path plumbing tests
decline Static decline response Policy/eligibility tests
agent-runner Local persona via agent-runner.js Cheap local advisory, subject to claim gate
gemini-review Cloud strong-model advisory via Gemini CLI Bounded strong review; keep prompts evidence-based and cost-limited

Claim-gate override

--claim-gate-off passes COMPANY_MESH_CLAIM_GATE=off to responder execution. Treat this as a break-glass/debug option only.

Use it only when all are true:

Do not use --claim-gate-off to make blocked agent-runner responses look valid. If claim gate blocks a response, prefer adding concrete evidence paths/pasted evidence or returning PARTIAL/BLOCKED honestly.

Evidence model

A run writes:

Evidence should be treated precisely:

Known gotchas

Current verified status

Completed evidence exists for the initial Company Mesh workflow runner smoke:

The next maturity step is a bounded non-smoke workflow using gemini-review or agent-runner against a real evidence artifact, with cost capped and no production mutation.

Slack Bot Runbook

Service: Slack Bot

Label: com.john.slack-bot
Tier: P1 (Critical)

What It Does

Claude-powered Slack bot that listens via Socket Mode and responds to messages. Uses an adapter registry (groq, claude-api, claude-cli, ollama) to select the best AI backend. Maintains conversation history per channel.

Dependencies

AI Tier-Routing — Which Model Answers

Updated: 2026-06-02 (MC #102825 fix)

The bot's reply engine is ~/system/tools/comms-responder.js using an adapter registry. Adapters run by priority (lower number = tried first):

PriorityAdapterModelPurpose
5groqllama-3.1-8b-instantFast fallback for tool-less/voice/trivial messages (~100ms, free tier)
10claude-apiSonnet (with tools) or Haiku (tool-less)Smart conversational messages, tool-use
20claude-cliSonnetCLI fallback
30ollamaqwenLocal fallback

THE RULE (Fix MC #102825, 2026-06-02)

Groq adapter SKIPS when options.tools?.length > 0 (returns null), because Groq cannot use Claude tool-use format. Tool-bearing conversational messages therefore fall through to claude-api → claude-sonnet-4-6.

Why this matters: slack-bot.js passes tools: TOOLS for every real channel/DM/mention message (lines 980-984, 1088-1089). Before the fix, groq intercepted EVERY message first with llama-8b (priority 5) and never reached Sonnet → weak replies. After the fix, groq skips tool-bearing messages → Sonnet answers with live data.

Routing Flow Examples

Tool-bearing message (e.g., "koliko otvorenih taskova imam?"):

1. Try groq (priority 5) → sees options.tools.length > 0 → returns null (SKIP)
2. Try claude-api (priority 10) → sees tools present → uses claude-sonnet-4-6 ✓

Tool-less message (e.g., voice, "ping"):

1. Try groq (priority 5) → no tools → executes with llama-3.1-8b-instant ✓ (fast path)

Fallback Chain Integrity

All adapters remain registered. No adapter was removed.

Cost & Latency Trade-off

Verification

To verify which adapter answered a message:

tail -50 ~/system/logs/comms-responder.log | grep "success"

# Tool-bearing message should show:
# [claude-api] success | model: claude-sonnet-4-6

# Tool-less message should show:
# [groq] success | model: llama-3.1-8b-instant

Common Failures & Fixes

Failure 1: "Invalid token" or "not_authed"

Symptoms: Bot fails to connect, log shows authentication error
Cause: SLACK_BOT_TOKEN or SLACK_APP_TOKEN expired or invalid
Fix:

# Check tokens in plist
cat ~/Library/LaunchAgents/com.john.slack-bot.plist | grep TOKEN

# Get new tokens from api.slack.com
# Update plist with new tokens
nano ~/Library/LaunchAgents/com.john.slack-bot.plist

# Reload plist
launchctl unload ~/Library/LaunchAgents/com.john.slack-bot.plist
launchctl load ~/Library/LaunchAgents/com.john.slack-bot.plist

Failure 2: "WebSocket connection failed" or "Socket Mode error"

Symptoms: Bot starts but cannot receive messages
Cause: Slack Socket Mode disabled or network connectivity issue
Fix:

# Check Socket Mode is enabled in Slack app settings (api.slack.com)
# Verify app-level token has connections:write scope

# Test network
ping slack.com

# Check logs for reconnection attempts
tail -50 ~/system/logs/slack-bot-error.log | grep -i socket

# Bot auto-reconnects, wait 30s or restart
launchctl restart com.john.slack-bot

Failure 3: "Claude API error" or "rate limit exceeded"

Symptoms: Bot receives messages but doesn't respond
Cause: Anthropic API down or rate limited
Fix:

# Check if ANTHROPIC_API_KEY is set
env | grep ANTHROPIC

# Test Claude API manually
curl -H "x-api-key: $ANTHROPIC_API_KEY" https://api.anthropic.com/v1/models

# Bot falls back to CLI if API fails
# Check if Claude CLI is available
which claude

# No immediate action needed, bot handles fallback automatically

Failure 4: "state file corrupted"

Symptoms: Bot crashes on start with JSON parse error
Cause: Corrupted slack-bot-state.json
Fix:

# Check state file
cat ~/system/config/slack-bot-state.json

# If corrupted, reset state (loses conversation history)
echo '{"channels":{}}' > ~/system/config/slack-bot-state.json

# Restart
launchctl restart com.john.slack-bot

Restart Procedure

launchctl unload ~/Library/LaunchAgents/com.john.slack-bot.plist
sleep 2
launchctl load ~/Library/LaunchAgents/com.john.slack-bot.plist

Verification

# Check running
launchctl list | grep slack-bot

# Check logs for connection
tail -20 ~/system/logs/slack-bot.log | grep -i connected

# Test bot in Slack
# Send message to bot channel, should respond

# Test backend connection
node ~/system/tools/slack-bot.js --test

Log Analysis

# Standard output (includes message activity, responses)
tail -50 ~/system/logs/slack-bot.log

# Errors (auth failures, API errors)
tail -50 ~/system/logs/slack-bot-error.log

# Look for connection status
grep -i "connected\|authenticated" ~/system/logs/slack-bot.log | tail -5

# Look for API errors
grep -i "api error\|rate limit" ~/system/logs/slack-bot-error.log | tail -10

Escalation

If restart doesn't fix:

  1. Verify Slack app tokens are valid (api.slack.com)
  2. Check Socket Mode is enabled in app settings
  3. Test Anthropic API key if using API backend
  4. Verify @slack/bolt npm package is installed
  5. Review state file for corruption
  6. Check Slack workspace status (status.slack.com)

Daemon Fleet — dr-sync & tldr-watch fix (MC #104330)

MC #104330 — fleet-watchdog alert resolution

Alert: [FLEET-WATCHDOG] 2026-06-25T06:36:57Z — CRITICAL: 2 daemons in failed state: com.john.dr-sync, com.john.tldr-watch

Root cause 1 — com.john.dr-sync (rsync exit 20)

The rsync exclude pattern *.bak does not match backup files named *.bak-<suffix>. An 18G stale backup mission-control.db.bak-pre-p2p-correction-20260529 (live db is 35M) was being rsynced to the mac-mini every 6h; the oversized transfer kept getting interrupted (exit 20), so the databases target failed (8/9 success) and the daemon exited non-zero.

Fix: ~/system/daemons/dr-sync.sh — added --exclude=*.bak-* and --exclude=*.bak[0-9]*.

Proof:

Root cause 2 — com.john.tldr-watch (exit 2)

Not a crash. tldr-watch is a health-monitor that exits 2 BY DESIGN when verdict=FAIL (script lines 119-122), and it owns its own alert path (#exec Slack + HiveMind intel). The fleet-watchdog only whitelisted exit 1/256, so tldr-watch's issue-found exit 2/512 was misclassified as a failed daemon.

Fix: ~/bin/daemon-fleet-watchdog.sh — added com.john.tldr-watch to EXIT1_NORMAL and extended allowed issue-found codes to 1/2/3 (+ launchd-encoded 256/512/768).

Proof: reclassification against live daemon-fleet-status.json → tldr-watch no longer critical.

End-to-end verification (L2+)

fleet-watchdog run 2026-06-26T08:46:40Z:

Follow-ups (separate, non-blocking)

  1. Disk hygiene: 18G stale backup still on disk (96% full / 42G free). Recommend CEO-approved deletion of mission-control.db.bak-pre-p2p-correction-20260529. Not deleted unilaterally (irreversible, not self-created).
  2. TLDR pipeline dormant: tldr-watch's FAIL is real — actionizer produces 0 insights/0 tasks daily, db counts static at 620,8,612,8 since ≥06-23. Decide: revive or retire tldr-briefing/actionizer.

LightRAG Recovery — Session 6c3cc7f0 Post-Mortem (2026-06-29)

LightRAG Recovery — Session 6c3cc7f0 Reconciliation

DocType: Incident Post-Mortem + Reconciliation Session: 6c3cc7f0-ff37-46c3-99ab-952b75a451b8 (2026-06-29) Documented by: FlowForge reconciliation pass (2026-06-30) MC Task: #104528 LightRAG drain + FORGE migration eval Incident Memo: incident_lightrag_403_extraction_and_chunkvdb_corruption_2026-06-29.md

1. Context: The "Do Not Touch" Rule

MEMORY.md contains the standing rule:

"LightRAG FIXED not dead — never re-diagnose/touch neo4j (project_lightrag_fixed_2026-06-12)."

Session 6c3cc7f0 did run approximately 26 az vm run-command invoke calls against vm-alai-lightrag (RG-ALAI-LIGHTRAG). See Flag A below for the alignment assessment.

2. Initial State (verified from transcript)

Parameter Value
VM vm-alai-lightrag, RG-ALAI-LIGHTRAG, public IP 20.240.61.67
Compose dir /home/alai-admin/lightrag/
LightRAG version api_version: 0154, core: 1.3.4
LLM_BINDING_HOST (pre-session) https://ollama.basicconsulting.no
EMBEDDING_BINDING_HOST (pre-session) http://ollama:11434 (correct, already internal)
LLM_MODEL llama3.1:8b
Internal ollama models llama3.2:1b, bge-m3:latest — llama3.1:8b was NOT present
processed 19,236
pending 40,559
failed 8–10
pipeline_busy false (stuck; pipeline not retrying)

3. Root Cause Chain (Three Links)

The session established the following confirmed chain of failures, documented in the incident memo:

.env line 10 pointed to https://ollama.basicconsulting.no, a leftover from before the domain migration to alai.no (pre-2026-05-17). The compose default in docker-compose.yml line 39 was ${LLM_BINDING_HOST:-https://ollama.basicconsulting.no}.

ollama.alai.no is served by cloudflared tunnel "mattermost" (id 3315a609-7934-45c5-ad0c-56d86d16374d). Its origin rule was pointing to http://10.0.0.2:11434 (FORGE Ollama). FORGE switched from Ollama to MLX (port 11435) on 2026-06-25. The Ollama :11434 service on FORGE was dead. The tunnel needed repointing to http://localhost:11434 (ANVIL local Ollama, which does have llama3.1:8b + bge-m3).

Link 3 — Ollama DNS-Rebind Guard (the actual 403 killer)

Ollama returns HTTP 403 to any request whose Host header is not localhost or 127.0.0.1. cloudflared by default forwards the original Host: ollama.alai.no header to the origin. This means every LightRAG extraction call received a 403 from ollama regardless of CF Access config.

Proof in transcript: curl -H 'Host: ollama.alai.no' localhost:11434/api/tags = 403 vs curl -H 'Host: localhost' localhost:11434/api/tags = 200.

Lesson (durable): cf-cache-status: DYNAMIC on a 403 response means the origin returned it — Cloudflare Access was NOT the gating layer. Do not assume 403 content-length:0 server:cloudflare means Access blocked it.

4. What the Session Did — Chronological

Step 1: Diagnosis

Ran az vm run-command invoke to read container state, docker ps, LightRAG logs, .env contents, and docker exec lightrag env. Confirmed extraction LLM host and 403 pattern.

Step 2: Wrong Fix — Internal Ollama (MY MISTAKE #1)

Pulled llama3.1:8b to the internal ollama container (docker exec ollama ollama pull llama3.1:8b, 4.9 GB). Edited .env to set LLM_BINDING_HOST=http://ollama:11434. Ran docker compose up -d lightrag.

Why this was wrong: The VM is 2 vCPU / 7.8 GB RAM / no GPU. This architecture is documented in project_lightrag_topology_fix_2026-06-03 — the LLM is remote BY DESIGN because the VM cannot host an 8B model. The result was OOM-thrash: ollama logged "llm server loading model" in a loop, never ready, ~195 MB RAM remaining. Zero docs drained.

Step 3: vdb_chunks.json Corruption (MY MISTAKE #2)

Ran docker compose up -d lightrag to recreate the container with the new .env (without a graceful stop first). The default stop timeout (10 seconds) sent SIGKILL mid-flush.

Effect: vdb_chunks.json was truncated from 132 MB to 48 MB. The matrix (chunk embeddings) was lost. On next start, nano_vectordb.load_storage raised JSONDecodeError → container entered crash-loop.

The large vdbs (vdb_entities.json 516 MB, vdb_relationships.json 759 MB) survived only by write-order luck.

Lesson (repeat of feedback_no_restart_prod_for_cosmetic_2026-06-04):

This container's graceful stop requires -t 180 (measured flush time: 3 minutes 4 seconds).
Never use default-timeout restart or docker compose up -d without a prior docker compose stop -t 180.
Verify backup integrity before any container lifecycle operation on this VM.

Step 4: Recovery from Backup

Source: ~/system/backups/lightrag/20260628-040003/lightrag-data.tar.gz (weekly cron snapshot)

Content: Valid vdb_chunks.json — 132 MB, JSON valid, matrix present, 17,234 data entries, content dated Jun 7 (consistent with entity/relationship vdbs frozen at that date).

Recovery sequence:

  1. Extract vdb_chunks.json locally from tarball.
  2. Upload to Azure Blob: plockfrontstaging/lightrag-backup/restore-20260629/vdb_chunks.json.
  3. Generate SAS URL (short TTL).
  4. On VM via az vm run-command: docker compose stop -t 180 (graceful, 3 min 4 sec measured).
  5. On VM: curl <SAS_URL> into volume mount /mnt/docker-data/docker/volumes/lightrag-data/_data/vdb_chunks.json.
  6. sha256 verify: b24174… (matched local extracted file).
  7. Update .env to correct LLM host (see step 5 below).
  8. docker compose up -d.

Verified result: t=80s RestartCount=0, Health=healthy. nano-vectordb loaded: vdb_chunks.json 17234 data — clean load, no JSONDecodeError. Available memory recovered from ~195 MB to 4.1 GB.

Step 5: Correct Fix — Cloudflared Tunnel + httpHostHeader

Critical infra fact (discovered in session): The tunnel is dashboard-managed (remote config). Editing /home/alai-admin/cloudflared/config.yml locally does nothing — cloudflared pulls remote config and logs "Updated to new configuration version=N". cloudflared tunnel ingress rule <url> reads the LOCAL file and misleads.

Fix path: CF API:

CF Access (Red Herring): IP-bypass policy already included VM 20.240.61.67/32 in BOTH the exact ollama.alai.no app (id bdc17e6a) and the *.alai.no wildcard app (id cd7cf0f0). CF Access was NOT the gating layer.

Verified after fix: VM → ollama.alai.no/api/tags = 200. /api/chat llama3.1:8b = 200 "OK". LightRAG logs showed "Merge N/E entities+relations". Zero new 403 errors.

5. Health Counts at Session Handoff

Taken from final docker exec lightrag python3 health probe in transcript:

Metric Value
llm_binding_host https://ollama.alai.no
embedding_binding_host http://ollama:11434
llm_model llama3.1:8b
processed 19,237 (incremented from 19,236 during session)
pending ~40,563–40,564
failed 8
processing 2
pipeline_busy True
RestartCount 0
403 errors post-fix 0

Note: processed incremented by 1 (+180s observation) after the correct fix was applied, confirming the pipeline was draining.

6. MC #104528 — Blocker Detail

Task: LightRAG drain + FORGE migration eval
Status: blocked (awaiting_forge — GOTCHA doc missing, /prompt-forge 104528 required before unblock)

Stated blocker in MC description: "add Cloudflare Access Bypass policy for VM egress IP 20.240.61.67/32 to ollama.alai.no app (Zero Trust, account d0ac2afb6bb5b298723b85a114151a04)"

Reconciliation note: The MC task description reflects the diagnosis made early in the session before the DNS-rebind guard was identified as the real root cause. Per the incident memo and transcript evidence, the CF Access bypass was already in place for 20.240.61.67. The actual fix (httpHostHeader + tunnel repoint) was applied and verified working within the same session. The stated MC blocker may now be resolved. This should be re-verified before the next session touches #104528.

Open sub-tracks in #104528:

  1. Evaluate migrating LightRAG + neo4j + ollama to FORGE (Mac Studio M3 Ultra 256 GB, OrbStack) to remove cloud/tunnel fragility. Prior reason to leave Mac: Docker Desktop crashed 3x on 2026-04-18. Counter: OrbStack removes that SPOF.
  2. Build LightRAG idle-watchdog: re-trigger pipeline when pending>0 && pipeline_busy=false for >N minutes. Without this, any future CF/tunnel blip will re-stick the 40k backlog indefinitely.
  3. Proveo validation task required per ZAKON PLAN before any drain claim.
  4. Skillforge BookStack documentation — this page closes that gap.

7. Flags

Flag A — "Do Not Touch" Rule Assessment {#flag-a}

The MEMORY.md rule states: "LightRAG FIXED not dead — never re-diagnose/touch neo4j (project_lightrag_fixed_2026-06-12)."

Assessment: The session's VM access was triggered by a real new incident (40k docs stuck pending, extraction 403 for an unknown period). The "FIXED" tag in MEMORY.md referred to a prior false-negative health-check (2026-06-12), not to permanent immunity from new incidents. The neo4j volumes were not touched. The touch was operationally justified.

However: The session made two compounding mistakes (internal-ollama wrong fix + non-graceful recreate) that would not have occurred if the topology documentation (project_lightrag_topology_fix_2026-06-03) had been read before acting. The "do not touch" rule exists partly to prevent exactly this class of well-intentioned damage. Standing rule recommendation: before any az vm run-command that modifies LightRAG container state, read the topology memo and the backup runbook first.

Verdict: Rule spirit violated (touch caused damage). Rule letter partially applies — the rule's "neo4j" qualifier was respected; the broader "do not re-diagnose" guidance was not.

Flag B — CF Access Service Token Plaintext Exposure {#flag-b}

During transcript tool result output (a Read call on lightrag.js source), the following values were printed in plaintext:

"lightrag.cf_access_client_id": "4248b2c109e87e09faf3fb82a90eeafd.access"
"lightrag.cf_access_client_secret": "[REDACTED — do not reprint]"

These are CF Access service token credentials for lightrag.alai.no. They were embedded in an alai_config block in the source file and were printed verbatim in the session transcript (file 6c3cc7f0-ff37-46c3-99ab-952b75a451b8.jsonl).

Risk: The transcript file is readable on-disk at ~/.claude/projects/-Users-makinja/. Any process or agent with filesystem access can read these values.

Recommended action: Rotate the CF Access service token for lightrag.alai.no (generate a new CID/CSEC pair in Cloudflare Zero Trust → Access → Service Auth). Update alai_config / Bitwarden. Sub-agent prompts should explicitly prohibit echoing secrets in output.

8. .env Configuration State at Handoff

File: /home/alai-admin/lightrag/.env

Key Value
LLM_BINDING_HOST https://ollama.alai.no
LLM_MODEL llama3.1:8b
EMBEDDING_BINDING_HOST http://ollama:11434

Backups on VM: .env.bak-pre-internal-llm-* (the wrong internal-ollama fix), .env.bak-pre-localembed-20260603 (older).

9. Architecture Notes (Durable)

Reconciliation completed 2026-06-30. No VM access performed during this documentation pass.

AI PR Review (Azure DevOps)

AI PR Review (Azure DevOps)

Status: LIVE on QODY (PR #76, Build 334). Bilko ready (PR #77, paused by CEO).
Engine: Gemini 2.5 Flash (REST API, pluggable architecture)
Mode: Comment-only (non-blocking, never fails merge)
Evidence: ~/system/evidence/105098/

1. What It Is

AI-powered PR review system for Azure DevOps repositories, equivalent to GitHub Copilot PR review functionality (which does not exist for Azure DevOps). Provides automated code review comments on pull requests using Gemini 2.5 Flash.

Why we built it: GitHub Copilot review is GitHub-only. Azure DevOps marketplace AI review extensions require sending our code to third-party vendors. This solution keeps control in-house while providing automated review feedback.

Key Features

2. Architecture

Components

  1. Script: tools/ai-pr-review.mjs (plain Node.js ≥18, zero npm dependencies — built-in fetch + git)
  2. Pipeline job: ai_pr_review in CI_Gates stage of azure-pipelines.yml
  3. LLM: Gemini 2.5 Flash via REST API (generativelanguage.googleapis.com, header x-goog-api-key)
  4. Azure DevOps APIs: PR iterations, threads, git refs

Data Flow

PR created/updated
   ↓
Branch Policy Build Validation triggers pipeline (NOTE: YAML pr: block is IGNORED on Azure Repos)
   ↓
ai_pr_review job runs (condition: Build.Reason = PullRequest)
   ↓
GET .../pullRequests/{id}/iterations → latest iteration SHA refs
   ↓
git fetch + git diff (commonRefCommit..sourceRefCommit)
   ↓
Diff filtering (exclude lock/generated/binary, 250KB limit, max 40 files)
   ↓
Gemini API call (one shot per run)
   ↓
GET .../threads (check for existing marker)
   ↓
POST new thread OR PATCH existing thread
   ↓
Build completes (success, never blocks merge)

Idempotency Mechanism

First line of every review comment: <!-- AI-PR-REVIEW:v1 -->
On each run:

  1. Fetch all PR threads
  2. Search for marker in thread comments
  3. If found: PATCH that thread (update in place)
  4. If not found: POST new thread

Result: 3 builds on same PR = 1 thread (verified live on QODY PR #76: builds 332/333/334 → thread id 115).

Fail-Safe Design

Every external operation (network, git, API) has timeout via AbortSignal.timeout. On any failure:

Draft PRs are automatically skipped (check SYSTEM_PULLREQUEST_ISDRAFT).

3. Setup for NEW Repository

Prerequisites

Step-by-Step Setup

A. Copy Script

  1. Copy tools/ai-pr-review.mjs from QODY or Bilko repo to your repo 
    # From QODY:
cp ~/business/ALAI-Holding-AS/products/qody/tools/ai-pr-review.mjs <your-repo>/tools/

B. Add Pipeline Job

  1. Edit azure-pipelines.yml, add this job to your CI_Gates stage (or create stage if none): 
    stages:
  - stage: CI_Gates
    displayName: 'CI Gates'
    jobs:
      # ... existing jobs ...
      
      - job: ai_pr_review
        displayName: 'AI PR Review (Gemini, comment-only)'
        condition: eq(variables['Build.Reason'], 'PullRequest')
        continueOnError: true
        timeoutInMinutes: 10
        pool:
          name: your-pool-name  # e.g., bilko-selfhosted, qody-selfhosted, or vmImage: ubuntu-latest
        steps:
          - checkout: self
            fetchDepth: 50  # CRITICAL: need history for git merge-base, NOT fetchDepth: 1
            persistCredentials: true  # CRITICAL: for git fetch in script
          - script: node tools/ai-pr-review.mjs
            env:
              SYSTEM_ACCESSTOKEN: $(System.AccessToken)
              GEMINI_API_KEY: $(GEMINI_API_KEY)
              REVIEWER_ENGINE: 'gemini'
              REVIEWER_MODEL: 'gemini-2.5-flash'

C. Configure Secret Variable

  1. Option A (Recommended): Azure Key Vault variable group 
    az pipelines variable-group create \
  --organization https://dev.azure.com/alai-holding \
  --project <project-name> \
  --name 'AI-Review-Secrets' \
  --authorize true \
  --variables GEMINI_API_KEY=<paste-key-here>
    Then reference in YAML: variables: - group: AI-Review-Secrets
  2. Option B: Pipeline-level secret variable via UI
    • Go to Pipeline → Edit → Variables → New variable
    • Name: GEMINI_API_KEY
    • Value: (paste from ~/system/config/secrets/gemini.json)
    • Check "Keep this value secret"

D. Set Repository Permissions

  1. Grant Build Service permission to post PR comments: 
    # Via Azure CLI:
az devops security permission update \
  --organization https://dev.azure.com/alai-holding \
  --project <project-name> \
  --subject "<project-name> Build Service (alai-holding)" \
  --token "repoV2/<project-id>/<repo-id>" \
  --allow-bit 16384  # Contribute to pull requests

    Or via UI:
    • Project Settings → Repositories → [your repo] → Security
    • Search for "[Project Name] Build Service (alai-holding)"
    • Set "Contribute to pull requests" = Allow

E. Configure Branch Policy

  1. CRITICAL: YAML pr: trigger block is IGNORED on Azure Repos (only works for GitHub/Bitbucket). PR builds require Branch Policy. 
    # Via Azure CLI:
az repos policy build create \
  --organization https://dev.azure.com/alai-holding \
  --project <project-name> \
  --repository-id <repo-id> \
  --branch main \
  --build-definition-id <pipeline-definition-id> \
  --display-name 'PR Build Validation' \
  --queue-on-source-update-only true \
  --manual-queue-only false \
  --blocking false  # Non-blocking for comment-only review

    Or via UI:
    • Project Settings → Repositories → [your repo] → Policies → Branch Policies → [main/master]
    • Build Validation → Add build policy
    • Select your pipeline
    • Policy requirement: Optional (non-blocking)
    • Trigger: Automatic

F. Test

  1. Create a test PR with a trivial change (e.g., add comment, fix typo)
  2. Verify:
    • Build triggers automatically
    • ai_pr_review job appears in pipeline
    • Bot posts a comment thread (look for marker at top)
    • Push another commit to same PR → same thread updates (footer shows new timestamp + build number)
    • Merge is NOT blocked

4. Troubleshooting

Problem: 403 Forbidden on POST threads

Symptom: Job log shows "Error posting review: 403"
Cause: Build Service lacks "Contribute to pull requests" permission
Fix: Step D above (set allow-bit 16384 on repoV2 token)

Problem: "could not read Password for 'https://dev.azure.com'"

Symptom: git fetch fails with credential error
Cause: persistCredentials: false on checkout (default in some templates)
Fix: Explicitly set persistCredentials: true in checkout step, AND add auth header: 

git -c http.extraheader="AUTHORIZATION: bearer $SYSTEM_ACCESSTOKEN" fetch origin $SHA
(Script already does this, but verify checkout step has persistCredentials: true)

Problem: Pathspec syntax error in git diff

Symptom: fatal: pathspec ':(exclude)package-lock.json' did not match any files
Cause: Passing pathspec through shell as string instead of argv array
Fix: Use execFileSync with array args (already implemented in script): 

execFileSync('git', ['diff', sha1, sha2, '--', ':(exclude)*.lock'])
// NOT: execSync(`git diff ... :(exclude)*.lock`)  ← shell parsing breaks on parens

Problem: Old PR has no review / "Policy not evaluated"

Symptom: Policy shows "Not configured" or review never appears
Cause: Branch policies apply only to PRs created/updated AFTER the policy is configured
Fix: Push an empty commit to the PR branch to re-evaluate: 

git commit --allow-empty -m "Trigger policy evaluation"
git push

Problem: Job not appearing on PR builds

Symptom: ai_pr_review job missing from pipeline run
Possible causes:

  1. No branch policy configured → YAML pr: block does NOT work on Azure Repos. See step E above.
  2. Condition not met: Verify condition: eq(variables['Build.Reason'], 'PullRequest') in YAML
  3. Wrong branch: Policy may be configured for main but PR targets develop

Problem: Script times out / no comment posted

Symptom: Job runs 10 minutes then cancels
Causes:

Debug: Check job log for last script output before timeout. Adjust timeoutInMinutes if needed (default 10).

Problem: Review quality is poor / too verbose

Tuning options:

5. Known Gaps & Roadmap

v1.0 (Current — LIVE on QODY)

v1.1 (Planned — MC #105102)

v2.0 (Future)

6. Live Deployments

QODY

Bilko

7. Security Notes

8. Key Lessons Learned

Lesson 1: YAML pr: Block Ignored on Azure Repos

Azure DevOps YAML pr: trigger blocks work ONLY for GitHub/Bitbucket repos, NOT Azure Repos. PR builds require explicit Branch Policy → Build Validation configuration.

Cost: 3 build iterations on QODY to discover this (policy cfg id=2 created, then all old PRs needed empty commits to re-evaluate).

Lesson 2: Git Pathspec Must Use execFileSync Argv

Pathspec syntax :(exclude)pattern breaks when passed through shell string (execSync) due to parentheses. ALWAYS use execFileSync with argv array for git commands.

Lesson 3: persistCredentials Required for Git Fetch

Checkout step defaults to persistCredentials: false in some templates, breaking git fetch in job. Explicit persistCredentials: true + http.extraheader auth required.

Lesson 4: Build Service Permission Non-Obvious

"Contribute to pull requests" (bit 16384) not granted by default to Build Service identity. First-run 403 error inevitable without pre-setup. Now part of standard checklist.

Last updated: 2026-07-09 | Author: Skillforge (John orchestration) | MC #105101

Memory Retrieval Eval Harness

Memory Retrieval Eval Harness — MC #105224

Purpose

Regression-test discover.js memory "<query>" retrieval quality with a labeled set of real incident queries mapped to known memo files.

Artifacts

How to run

node ~/system/tools/memory-eval-harness.js run \
  --test-set ~/system/specs/memory-retrieval-eval/test-set-v1.json \
  --output ~/system/evidence/memory-eval-$(date +%Y%m%d-%H%M%S)

Dry-run schema validation:

node ~/system/tools/memory-eval-harness.js run \
  --test-set ~/system/specs/memory-retrieval-eval/test-set-v1.json \
  --dry-run

Durable P0-A regression test (default/top-N prefixes, argument validation, fail-closed privacy, and metric recomputation):

node ~/system/tools/tests/memory-retrieval-p0a.test.js

Metrics

Normal discover.js memory "<topic>" calls remain capped at top 3 for backward compatibility. The eval harness explicitly uses --top 20, which changes only truncation depth, not scoring or ordering.

Baseline run — 2026-07-10 (historical, before P0-A)

Fresh verification output: ~/system/evidence/memory-eval-105224-verify-20260710/

P0-A verification — 2026-07-21

Evidence: ~/system/evidence/memory-eval-p0a-20260721/

The apparent Recall@3 change versus the historical baseline reflects corrected handling of documented fail-closed exit code 1 and evaluation over one top-20 candidate list. The --top path changes only truncation, not scoring or ordering.

Privacy-safe retrieval telemetry

Every discover.js memory invocation appends aggregate metadata to ~/system/logs/discover-queries.jsonl (mode 0600). It records timestamp, query length/token count, requested top-N, result/confidence counts, latency, fail-closed state, and coarse error kind. It never records query text or hashes, memo filenames/content, secrets, or PII.

FORGE gpt-oss-120B Think-Tier Pilot — MC #105426

gpt-oss-120B MXFP4-Q8 Pilot — Final Report

MC: #105426
Date: 2026-07-12/13
Agent: AgentForge
Status: COMPLETE

Executive Summary

gpt-oss-120B-Q8 installed on FORGE MLX and benchmarked across 10 real ALAI tasks.

Result: 7/10 accuracy (70%), 11.8s avg latency — BEST accuracy in practical-latency class.

Recommendation: ADOPT for experimental think-tier (H/BLOCKER novel tasks) with boolean normalization processor.

Final Benchmark Results (10 Tasks)

Model Correct Accuracy Errors Avg Latency
gpt-oss-120b-Q8-raw 7/10 70% 0 11.8s
deepseek-r1-70b 7/10 70% 0 77.2s
gpt-oss-120b-Q8-fixed 6/10 60% 0 12.8s
qwen2.5-coder-32b 6/10 60% 0 7.8s
qwen2.5-7b 5/10 50% 0 7.0s

Key Finding: Format fix HURT accuracy (70% → 60%). Model prefers clean minimal prompts.

Installation

Failure Analysis

All 3 gpt-oss-120B failures share IDENTICAL root cause: FORMAT, not reasoning.

Actual reasoning quality: 10/10 (all verdicts logically sound).

Solution: Response processor to normalize boolean → "PASS"/"FAIL".

Routing Accuracy: PERFECT

7/7 dispatch tests correct (100%):

qwen2.5-coder-32b failed T10 (answered FlowForge instead of Finverge).

Tier Routing Recommendation

✅ ADD to Experimental Think-Tier

{
  "think-tier": {
    "model": "/Users/makinja/models/gpt-oss-120b-MXFP4-Q8",
    "endpoint": "http://10.0.0.2:11435/v1/chat/completions",
    "timeout": 30,
    "use_for": ["H", "BLOCKER"],
    "task_types": ["novel", "red-zone", "ambiguous-routing"],
    "response_processor": "normalize_boolean_verdicts"
  }
}

Required processor:

function normalize_boolean_verdicts(response) {
  if (typeof response.verdict === 'boolean') {
    response.verdict = response.verdict ? "PASS" : "FAIL";
  } else if (response.verdict === "true") {
    response.verdict = "PASS";
  } else if (response.verdict === "false") {
    response.verdict = "FAIL";
  }
  return response;
}

❌ KEEP Existing Defaults

✅ DEPRECATE

Resource Utilization

Cost-Benefit

Direct cost: $0 (local FORGE inference)
API savings: $1,100–1,800/year (vs Opus 4.8 for think tasks)
Opportunity cost: $7,300/year if overused on volume (11.8s latency adds up)

Verdict: Cost-justified for CRITICAL-PATH tasks where accuracy > speed.

Pilot Deployment Plan

Phase 1 (2 weeks):

Phase 2 (if successful):

Phase 3 (after 1 month):

Evidence Files

All in ~/system/evidence/105426/:

Conclusion

gpt-oss-120B is PRODUCTION-READY for limited think-tier use:

Recommendation: APPROVE pilot deployment with boolean normalization layer.

Azure litestream egress saga + backup policy (lokalno do plaćenih korisnika) — MC #105462

MC #105462 — Azure litestream ListBlobs egress: FINAL REPORT (za CEO)

Datum: 2026-07-13 | Sesija: john pid-7080 (nastavak; prethodne: c10520f4 team-lead + izvršna sesija pid-20439) Status AC-a: čist mjerni prozor počeo 2026-07-13T08:52:24Z — formalna AC provjera (ListBlobs <5GB/6h) moguća od 14:52Z (16:52 lokalno); 2h checkpoint od 10:52Z (12:52 lokalno).

1. Sažetak

Egress na alaibackups0ebb narastao do ~518 GB/dan (~370 NOK/dan). Root cause NIJE stari l0 graveyard nego litestream per-sync-tick ListBlobs: svaki sync tick lista cijeli ltx prefiks svog DB-a, a flywheel (~113K blobova, 68% egressa) i mission-control (~102K blobova, 23%) imali su 30s/1s intervale. Atribucija dokazana iz $logs storage-loga: exact IP + user-agent + call-rate == konfigurisani intervali (ATTRIBUTION-FINDINGS.md).

2. Šta je primijenjeno (Phase A — GOTOVO, verifikovano)

Zahvat Detalj Evidence
Sync intervali (1. restart 07:45:56Z) mission-control 1s→60s, flywheel 30s→600s restart-verification.md, litestream.yml.before/after-105462
Phase A extension (2. restart 08:52:24Z) 6 telemetrijskih DB → 300s; svih ostalih 56 DB <60s → 60s litestream.yml.before/after-phaseA-extension
Azure lifecycle 8d delete pravila proširena sa 9 na svih 64 DB prefiksa policy-after-phaseA-verified-live.json
Restart verifikacija Novi PID 11505, log potvrđuje svaki novi interval, status=ok, txid napreduje restart-verification.md

3. Incident tokom rada (kolizija sesija — sadržano, bez gubitka podataka)

Druga CEO-terminal sesija (c4a3b3f9, PID 27243, živa od 07-10) izvršila je STARE delete-batch korake iz originalnog opisa taska (otkazane u međuvremenu), bez claim lease-a — 2 talasa brisanja l0 blobova na 7 baza (do 92% blobova na agent-routing). Sadržavanje: SIGSTOP cijelog stabla PID 27243 + lease + STOP-DELETE notice u task opisu. Puni detalji: INCIDENT-unauthorized-deletes.md.

DR re-verifikacija poslije oba talasa: SVIH 7 BAZA RESTORABLE — NEMA GUBITKA PODATAKA. 5 baza puni restore + integrity_check=ok + exact row-count match; flywheel i mission-control preko snapshot-coverage dokaza (level-9 snapshot pokriva txid raspon ispod najstarijeg preživjelog l0). Follow-up preporuka: pun restore test za te 2 baze u maintenance prozoru (belt-and-suspenders).

3b. Kolizija #2 (kasno veče) — kontradiktorne CEO odluke iz dvije paralelne sesije, RIJEŠENO bez akcije

Oko 23:10Z, prije primjene team-lead-instruirane pune lokalne migracije, ova sesija je uočila da task record (mc.js show 105462) pokazuje status=paused i tekst "CEO ODLUKA 2026-07-13 11:00: FAZA B = B1 (pusti retention roll-off; B2 odbačen)" — direktno u suprotnosti s uputstvom primljenim ovim kanalom (puna lokalna migracija, uključujući mission-control). Dodatno, u istom evidence direktorijumu postojao je REPORT.md (16:56, tag "john pid-7080") s eksplicitnom preporukom "MC→local NE preporučujem". ps je potvrdio PID 7080 kao ŽIVU, aktivnu sesiju (elapsed 14+ h u tom trenutku).

Sesija je ZAUSTAVILA svaki dalji rad i eskalirala prije nastavka (poruka ~23:10Z), umjesto da nastavi na osnovu konfliktnih signala ili da jednostrano revertuje migraciju. Razrješenje (potvrđeno od team-lead-a): obje odluke su GENUINE — "11:00 B1" je iz PARALELNE sesije pid-7080 (CEO-ov drugi terminal, otvoren 10:50), donesena PRIJE dokaza da throttling ne radi (Phase A

Ključna razlika od Kolizije #1 (sekcija 3): ovdje NIJE bilo destruktivne akcije niti rizika gubitka podataka — samo dvije verzije istine u task recordu i evidence direktorijumu. Sesija pid-7080 nije dirana (nije njena da se ubija/koordinira — team-lead preuzima taj kontakt s CEO, isto kao za jutrošnji c4a3b3f9). Task record ažuriran (23:13:55) s važećom odlukom, "NE REVERTOVATI bez nove CEO odluke" klauzulom i napomenom o paralelnoj sesiji, tako da sesija pid-7080 (kad se sljedeći put probudi/čita bazu) vidi TAČNO stanje umjesto zastarjelog.

Zajednički nalaz obje kolizije danas — najjači argument za #105241 (PI intake/claim disciplina): dva odvojena incidenta u istom danu, oba korijenski uzrokovana time što paralelne John/CEO sesije rade na istom MC tasku bez claim-lease vidljivosti ili sinhronizacije task-recorda u realnom vremenu — prva rezultovala stvarnim brisanjem podataka (sadržano, bez gubitka), druga rezultovala kontradiktornim infrastrukturnim odlukama u istom konfiguracionom fajlu (sadržano, bez štete jer je novija odluka bila ispravna). #105241 trenutno adresira samo PI-orchestrator auto-claim slučaj (3 dokumentovana primjera); ovaj dan pokazuje da isti problem postoji i između DVIJE John/CEO sesije bez PI posrednika — claim-lease enforcement (već predložen kao #105465 iz Kolizije #1) trebao bi pokriti OBA slučaja, ne samo PI-vs-John nego i John-vs-John.

4. Trenutno stanje egressa

Očekivana mehanika oporavka: egress ∝ (broj tickova × veličina liste). Intervali su odmah srezali broj tickova (MC 60×, flywheel 60×, 54 DB-a 60×). Veličina liste pada kako litestream retention (flywheel 72h, MC 168h) + Azure lifecycle 8d roll-off smanje broj blobova: flywheel novi tempo ≤144 blobova/dan (sa ~30s tempa), pa za 3 dana l0 count pada sa ~113K na stotine; MC za 7 dana sa ~102K na ~10K. Puni efekat: 3-7 dana, organski, bez ikakvog brisanja.

5. FAZA B — odluke za CEO

B1. ✅ CEO ODABRAO 2026-07-13 11:00: NIŠTA više ne dirati — pustiti retention roll-off. (B2 odbačen za danas.) Phase A + lifecycle već garantuju pad; jedini trošak je repni egress dok se blob countovi ne istope (grubo: nekoliko stotina GB kroz sedmicu, opadajuće). Nula DR rizika. Dnevna kontrola očitanja do AC PASS.

B2. Ubrzanje (opciono, tek nakon AC mjerenja): forsirati kompakciju/snapshot na flywheel+MC. litestream snapshot + niža retention privremeno → l0 se topi za sate umjesto dana. Ušteda ~200-400 NOK repnog egressa, ALI dira DR lanac dan nakon incidenta — tražim Proveo-verifikovan runbook i CEO GO prije izvršenja. NE preporučujem danas.

B3. Strukturno (poseban task, nije hitno): hot-churn telemetrijske baze (flywheel, trace/health-events) dugoročno ne pripadaju per-tick replikaciji — konsolidovani backup tier ili litestream noviji level-config. Predlažem M task nakon što AC prođe.

B4. Sudbina zamrznute sesije (CEO odluka — tvoj terminal): PID 27243 stablo je SIGSTOP (T state). Drži SP secret u plaintext cmdline-u (vidljivo u ps i SADA — provjereno ovom sesijom). Preporuka: SIGKILL stabla čim potvrdiš da u tom terminalu nema nespašenog rada, pa rotacija.

B5. SP rotacija alai-backup-writer (#105466, paused): secret kompromitovan kroz ps/transkripte. Prethodna sesija blokirana permission bound-om na az ad sp credential. Treba ili CEO az ad app credential reset ili Graph permission za service principal. VEZANO za B4 (rotacija prije kill-a je beskorisna dok frozen proces drži stari secret u memoriji — redoslijed: kill → rotate → update plist + BW).

5b. 2h checkpoint (12:54-13:40 lokalno) — NALAZI

Mjerenje (egress-clean-2h-checkpoint.tsv): ListBlobs ~17.5GB u čistom 2h prozoru (08:52-10:52Z) → pace ~52GB/6h vs AC <5GB/6h. Pozivi ~5.8K/30min (bilo 8.4K pre-fix — pad samo ~2×, ne 60×).

Atribucija pozivaoca — ZATVORENA: jedini proces koji priča s alaibackups0ebb je litestream PID 11505 na ovoj mašini (lsof 70×1s sampling: 100% litestream; sve transakcije OAuth=SP; FORGE ssh-verifikovan čist, Azure VM čist — backup cron 03:00, LaunchAgents čisti, nula rogue procesa). Oscilacija 09→10Z je litestream-ova vlastita (hourly retention pass + "behind replica" retry petlje), NE novi klijent.

Zašto AC danas neće proći: veličina list odgovora je ∝ broju blobova; flywheel (~113K) i MC (~102K) l0 blobovi se NE tope organski — u logu nula "l0 retention enforced" / "compaction complete" eventova za flywheel/MC (litestream retention za njih efektivno ne radi na ovolikim prefiksima). Roll-off zavisi od Azure lifecycle 8d pravila (postavljena danas 07:23Z; prva egzekucija tipično ≤24-48h). Realna prognoza AC PASS: 2026-07-14/15 nakon prve lifecycle egzekucije. Fix je i dalje ispravan — intervali su srezali tick-komponentu; ostaje count-komponenta koju čisti lifecycle.

NOVI DR NALAZ (H task #105490): lifecycle pravilo BackupRetentionRule je account-wide (bez prefixMatch!) s tierToArchive@30d — arhiviralo je L0 blobove 35 uspavanih DB-ova → litestream 409 BlobArchived retry petlja svakih 5min → tih 35 DB-ova NE replicira (tihi DR gap) (telemetry, escalations, companies, pipeline...). Fix zahtijeva CEO GO (dira lifecycle policy): scope-ovati pravilo + reinit malih replika. Detalji u tasku #105490.

5c. Pi checkpoint (13:24 lokalno / 11:24Z) — READ-ONLY

Artifact: CURRENT-STATUS-20260713T1124Z.md

Zaključak: task ostaje PARTIAL / čeka formalni 6h prozor u 14:52Z (16:52 CEST). Ne dirati više (blob delete i B2 i dalje STOP); B1 ostaje važeća CEO odluka.

5d. FORMALNO AC MJERENJE (16:56 lokalno) — REZULTAT

AC FAIL danas, očekivano: ListBlobs egress u formalnom 6h prozoru (08:52:24-14:52Z) = 58.88 GB vs AC <5GB/6h (egress-AC-6h-final.tsv). Napomena: prozor je miješanih režima — u 13:52Z upala je flywheel→local migracija (CEO-autorizovana u drugoj sesiji, 5c/restart-verification).

Ključni novi nalaz — dominantni lister je MISSION-CONTROL, ne flywheel: post-migracijski steady state (13:52-14:52Z, flywheel 100% van Azure-a) = ~3GB/15min ≈ 12GB/h, NEPROMIJENJENO vs prije migracije. Račun se poklapa: MC ~102K blobova ≈ 21 stranica × ~1MB po listingu, listano više puta po 60s ticku (multi-level plan) ≈ 12GB/h. Flywheel-ov udio u pozivima bio je već zanemariv poslije Phase A (600s = 6 lista/h); "flywheel 68%" iz jutarnje atribucije važio je za PRE-Phase-A režim (30s).

Preostala poluga = Azure lifecycle 8d na litestream/mission-control/ (litestream vlastiti retention za MC dokazano ne radi — 0 eventova). Većina MC-ovih 102K L0 blobova je starija od 8 dana → prva lifecycle egzekucija (tipično ≤24-48h od 07:23Z danas) briše bulk → očekivan pad ~10×, a post-Phase-A tempo punjenja je ≤1.440 blobova/dan. Prognoza AC PASS: 2026-07-14/15; re-mjerenje sutra ujutro (boot Next Steps).

Opcije ubrzanja (SAMO uz CEO GO, default ostaje čekanje po B1): B2-lite samo za MC (forsirani snapshot + prune starih generacija); MC→local NE preporučujem (MC je srž sistema — cloud DR mu je i svrha, za razliku od flywheel cache-a).

6. Otvorene stavke do zatvaranja taska

  1. ⏳ Formalna AC 6h provjera: 14:52Z (16:52 lokalno). AC: ListBlobs <5GB/6h. 2h i 13:24 checkpointi već pokazuju da AC vjerovatno neće proći prije lifecycle roll-offa.
  2. ⏳ CEO odluke otvorene: B4 (frozen sesija), B5 redoslijed rotacije. B1 je potvrđen 2026-07-13 11:00; B2 odbačen za danas.
  3. 📋 Follow-up taskovi: #105465 (claim-lease enforcement za destruktivne korake), #105466 (SP rotacija), #105490 (lifecycle archive DR gap), predloženi M task za pun restore test flywheel/MC + B3 strukturni.

Evidence index (sve u ~/system/evidence/105462/)

ATTRIBUTION-FINDINGS.md · INCIDENT-unauthorized-deletes.md · restart-verification.md · litestream.yml.before/after-* · policy-after-phaseA-verified-live.json · egress-.json · restore-verify-.{log,json} · l0-*.csv · raw-listings/ · storage-logs/

7. CEO POLICY ODLUKA 2026-07-13 ~21:55Z — SUPERSEDES sekciju 5 (B1-only)

CEO odluka (verbatim, primljena prvom rukom u sesiji c10520f4 ~21:55Z; zapisana u memo project_backup_policy_local_until_paying_customers_2026-07-13.md u 23:02): "Prebaci lokalno! Nemamo placene korisnika kad to dodje ide backup na azure do tada save money gdje mozemo."

Eksplicitno nadjačava odluku "11:00 B1, B2 odbačen" (sekcija 5) i preporuku iz 16:56 ("MC→local NE preporučujem", sekcija 5d). Nakon dokaza da mission-control i uz pun throttling proizvodi ~12GB/h, CEO je izabrao kompletan fix umjesto parcijalne mitigacije. Nije governance konflikt — evolucija odluke kroz dan s novim dokazima.

POLICY (na snazi): SVE sistemske SQLite baze (litestream.yml, 64 DB) repliciraju SAMO na lokalni disk (~/backups/litestream//, retention ~72h). DR troslojan: lokalna litestream replika (kontinuirano) + lokalni daily db-backup.sh (5d) + B2 offsite 6h (Backblaze, netaknut). Azure storage account alaibackups0ebb se NE briše — 8d lifecycle drenira l0, decommission odluka ~30d, account čuvan za reaktivaciju. Uslov reaktivacije: prvi PLAĆENI korisnik — tada se cloud replikacija vraća, uz prethodni litestream upgrade research (izbjeći full-prefix-per-tick listing).

8. Izvršenje (22:05-23:08Z)

Disk projekcija prije primjene: 63 preostale baze = 713.84MB × 3 = ~2.09GB (ispod 20GB triggera; sanity-check protiv flywheel realnog ratia 0.9x). Transform: awk skripta validirana litestream config parserom prije žive primjene (dry-run: 64/64 file, 0 abs). retention/check-interval/snapshot-interval očuvani per-DB.

Restart: config mtime 22:05:09Z, novi PID 87017, stari 42717 terminiran. Log: 0× "type=abs", 64× "type=file". Svih 63 novo-migriranih DB: očekivani tranzijentni "ltx file missing" prvi sync → svih 63 self-resolved svježim snapshotom u sekundama (63/63 "snapshot complete", 0 s consecutive_errors>1). litestream status: 64/64 "ok".

Disk stvarno: 927MB ukupno (44% projekcije). Najveći: flywheel 346MB, hivemind 220MB, knowledge 167MB, session-index 69MB.

B2 pokrivenost potvrđena ENUMERACIJOM (source-read oba joba — rclone-backup.sh nightly + offsite-backup.sh 6h — nefilterirani globovi nad ~/system/databases/; svih 64 DB fizički prisutno, nula MISSING).

9. FINALNO AC MJERENJE — DEFINITIVAN PASS

ListBlobs po satu: 19:08Z 11.71GB, 20:08Z 12.35GB, 21:08Z 12.25GB (baseline) → 22:08Z 0.0000470 GB (prvi potpuno čist sat). 5-min granularnost 22:08-23:03Z: 14 uzastopnih bucketa = 0 GB (šum ~20-25KB). AC (<5GB/6h): PASS za nekoliko redova veličine. Mehanizam: nula mrežnih poziva prema Azure za replica sync — eliminisana cijela klasa curenja, uključujući nerazjašnjeni 4-6x gap iz sekcije 5d (moot — "unexplained residual mechanism, made moot by full local migration").

10. Task record

MC #105462 record ažuriran 23:13:55 od john/c10520f4: važeća odluka + supersede + NE-REVERTOVATI klauzula. (Raniji pokušaj flowforge agenta ispravno blokiran actor-token gate-om.)

12. Disk drain (checkpoint 1 flag) — RIJEŠENO

35Gi pad (94Gi→59Gi) tokom popodneva NIJE bio litestream leak niti bilo šta vezano za ovaj task — dva puna LumisCare docker builda (uklj. --no-cache verify) → docker images/build-cache, gradle dependency download ~8.4GB, 27 APFS snapshota pinovalo obrisane blokove. Disk stabilan na ~61Gi poslije brzog prune-a od strane team-lead sesije, dok je ova sesija radila migraciju. Temeljitija higijena praćena kao #105584 (docker image prune -a, stari image-i 274/37GB triage, .gradle cache policy, APFS snapshot check, cilj <88% capacity, NEEDS REVIEW). Root cause dijagnoze uzrok (boot.sh je čitao pogrešan volume) praćen kao #105521, nezavisno od ovog nalaza. Zatvoreno — bez daljeg praćenja u ovom tasku.

13. backup-health-check.sh scheduling — RIJEŠENO

Skripta (ListBlobs egress alarm, izgrađena ranije danas) nikad nije bila raspoređena nigdje (potvrđeno pretragom launchctl/crontab/plist — 0 pogodaka). Dedicated LaunchAgent otpada (launchctl load permission-blokiran za agent sesije). Fix: piggyback na ~/system/daemons/offsite-backup.sh (6h kadenca, com.john.offsite-backup LaunchAgent), isti pattern kao postojeći LumisCare demo Postgres re-stop guard (MC #105489) u istom fajlu: perl -e 'alarm 180; exec @ARGV' -- bash <script> >> <log> 2>&1 || true — 3-minutni timeout, nikad ne obara backup job. Syntax-checked (bash -n, exit 0). Test-run izolovano (bez trigerovanja stvarnog B2 synca): health check je odradio pun ciklus, ulogovao 4 real nalaza (GitHub push stale, 3 Docker kontejnera ne rade na ANVIL-u — pre-postojeći uslovi, nevezani za ovaj task), Slack notifikacija poslata, combined exit code 0 (guard potvrđeno radi). ListBlobs check specifično: WARN "could not read metric" tokom testa — dijagnostikovano: Azure Monitor sa NULA ListBlobs saobraćaja izostavlja timeseries entry umjesto da vrati 0, pa upit ne nalazi ništa (graceful degradation, ne bug — isti upit je ranije danas radio ispravno kad je bilo stvarnog saobraćaja za mjerenje). Detalji: backup-health-check-piggyback-test.md.

14. Konsolidovani follow-upi

CEO reaktivacioni uslov (verbatim, na snazi)

"Prebaci lokalno! Nemamo placene korisnika kad to dodje ide backup na azure do tada save money gdje mozemo." Azure backup replikacija se REAKTIVIRA s prvim PLAĆENIM korisnikom. Reference: memo project_backup_policy_local_until_paying_customers_2026-07-13, MC task #105462 opis (ažuriran 23:13:55). Prije reaktivacije: obavezno istražiti litestream upgrade koji ne lista pun ltx prefiks na svaki sync tick (follow-up gore) — ponavljanje ovog leaka bez tog istraživanja je siguran ishod pri ponovnoj Azure replikaciji istog obima podataka.

Mail signal & [INBOX] flood — zašto je CEO signal bio strukturno mrtav (2026-07-24)

Mail signal & [INBOX] flood — zašto je CEO signal bio strukturno mrtav (2026-07-24)

MC taskovi: #106283 (status: paused, čeka Proveo živu validaciju) · #106289 (status: ready_for_review) · #106285 (status: in_progress) Ova stranica je DoD-gate zahtjev za sva tri — dokumentacija je jedina preostala stavka prije zatvaranja.

1. Simptom i kako je otkriveno

CEO gut-feeling test: "provjeri manuelno mail čitanje? koristi API vs naš alat — imam gut feeling da nešto ne radi."

Boot mail-signal filter (korak 2 u ~/.claude/CLAUDE.md / /Users/makinja/CLAUDE.md) vraćao je nula pogodaka svaku sesiju, iako je CEO u međuvremenu slao mailove. Tri nezavisna nalaza povezana u ovoj sesiji objašnjavaju zašto — dva strukturna uzroka (#106283, #106289) i jedna posljedica istog roditeljskog problema koja se manifestovala kao zaseban simptom (#106285).

2. Root cause #1 — \Seen je dijeljeno mutabilno stanje (MC #106283)

Glavna pouka stranice: IMAP \Seen flag nije validna baza za signal kad bilo koji drugi proces čita isti mailbox — prvi čitalac ga potroši. Signal mora doći iz vremenskog prozora nad DB-om (append-only ingest), ne iz flaga koji je dijeljeno mutabilno stanje.

3. Fix #106283 — DB time-window umjesto IMAP unseen

mail-boot-signal.sh prepisan:

Dokaz (prije/poslije):

STARO: account=john total_unread_scanned=0
NOVO:  accounts=ALL window_hours=72 total_scanned=170 new_matches=1 slack=posted ... state=ok
       [MAIL SIGNAL] alembasic@gmail.com (account alem) — ... (2026-07-22T08:11:49Z)
       [MAIL SIGNAL] alem@alai.no (account alem) — Fwd: PR ... (2026-07-21T18:05:43Z)

Cross-account pogodak dokazan uživo (AC1), dedupe potvrđen (dva uzastopna runa, drugi new_matches=0, AC3), staleness alarm okinut na agresivnom pragu i tih na normalnom pragu (AC5).

Razvojni bug vrijedan pomena: prvobitna verzija je koristila IFS=$'\t' za parsiranje redova iz DB-a — bash tretira tab kao IFS-whitespace pa kolabira uzastopne separatore kad je neko polje prazno; from_name je prazno u skoro svim redovima → sva polja poslije njega se pomjere ulijevo i subject se izgubi/pogrešno mapira. Fix: separator promijenjen na \x1f (unit separator, ne pojavljuje se u tekstu maila).

Evidence: ~/system/evidence/106283/ — ac1-cross-account-hit.log, ac3-dedupe.log, ac4-migration-no-flood.log, ac5-staleness-guard.log, ac6-doc-sync-diff.txt, final-mail-boot-signal.sh, gotcha-task-106283.md.

4. Root cause #2 — signal se davi u CI šumu (MC #106289)

Fix #106283 je bio tehnički tačan ali operativno beskoristan bez ovog drugog fixa: SIGNAL_PATTERN je sadržavao goli INVOICE (bez word-boundary), koji matchuje substring unutar hr-einvoice u Bilko Azure DevOps commit/PR subjektima (npr. fix(hr-einvoice): BT-23/BT-24 UBL conformance).

Korekcija broja (bitna za integritet, ne izostaviti): task-opis je tvrdio "samo 2 stvarna CEO maila" preostala. Živa provjera preostalih 16-11=5 redova pokazala je 4 zadržana pogotka, ne 2: 2 stvarna CEO maila (alembasic@gmail.com, alem@alai.no — webinar fwd) + 2 Apple "Fakturaen din fra Apple" (matchuju na faktura keyword, van scope-a ovog fixa, legitimno zadržani da se signal ne prefiltrira). 5. red (nejasno koji) nije naveden u verdict.md kao dio ni jedne kategorije — ostavljeno kako je zatečeno, nije ovog fixa scope.

Fix (~/system/tools/mail-boot-signal.sh):

SIGNAL_PATTERN='...|\bINVOICE\b|faktura'          # word-boundary
CI_SENDER_EXCLUDE_PATTERN='azuredevops@microsoft\.com|azure-pipelines@|azure-noreply|noreply@dev\.azure\.com'

Sender-exclude provjera dodana PRIJE keyword matcha u oba loop-a (linije 178 i 203 u trenutnoj verziji skripte). Production state fajl očišćen 16→4 zadržana pogotka (backup: seen-uids-BEFORE-cleanup.txt, rezultat: seen-uids-AFTER-cleanup.txt). Finalni produkcijski run: exit=0, new_matches=0, bez ponovnog paljenja 11 uklonjenih CI message_id-ova.

Evidence: ~/system/evidence/106289/ — AC1-ci-sender-before-after.txt, AC2-hr-einvoice-example.txt, ac3-isolated/, AC4-legit-invoice-still-matches.txt, AC6-production-dedupe-proof.txt, before-after.txt, seen-uids-BEFORE/AFTER-cleanup.txt, verdict.md.

Pouka: kad oživiš mrtav signal, odmah provjeri KO su pogoci. Nijedan acceptance-criterion u #106283 nije to pitao — defekt je izašao tek naknadnim ručnim mapiranjem message_id → pošiljalac protiv 1249 CI redova.

5. [INBOX] flood (MC #106285)

284 otvorena H-priority [INBOX] taska u mission-control.db, verifikovano direktnim SQLite upitom (ne CLI paginacijom):

SELECT COUNT(*) FROM tasks WHERE status='open' AND priority='H' AND title LIKE '[INBOX]%'  -- => 284

Sender breakdown (raw enumeracija, enumeration-284-full.csv):

149 + 90 = 239 = stvarni "notification flood" zbog kojeg je task otvoren.

Fix u ~/system/tools/inbox-watcher.js (isNewsletterOrNotification()): nova CI grana koja za azuredevops@microsoft.com / azure-pipelines@* / noreply@dev.azure.com vraća {isNoise:false, category:"ci_notification", priority:"M"} — snižava prioritet na M, NE guši signal (isNoise ostaje false, task se i dalje kreira, samo ne na H). Namjerno drugačije od self_digest guarda (2026-07-20, full suppress) jer "Build failed" jeste stvaran signal. createMcTask() proširen parametrom priority (default "H", sad prosleđen "M" za CI granu).

Acceptance replay (vm.runInContext nad živom funkcijom, ne ručni prepis logike):

CASE 1 (Build failed, azuredevops@microsoft.com) → {isNoise:false, category:"ci_notification", priority:"M"} — PASS
CASE 2 (regresija: self_digest i dalje suppressed) → PASS
CASE 3 (regresija: legit known-contact mail neizmijenjen) → PASS

Triage 284 → 239 zatvoreno, 45 ostalo otvoreno (9 KEEP-OPEN + 36 UNVERIFIED), enumeracijom, ne WHERE-pretpostavkom:

Kategorija Broj Odluka Dokaz
[Build failed] 96 CLOSE-STALE build history + PR status per task (88 superseded, 8 pojedinačno provjereno abandoned/merged/dormant)
[PR build failed] 1 CLOSE-STALE PR #218 abandoned
[Run stage approval pending] 49 CLOSE-STALE live _apis/pipelines/approvals: 0 pending u oba projekta
PR merge notifikacije 2 CLOSE-STALE PR #47/#170 status=completed
PAT rotation info 1 CLOSE-STALE informativno, poznat događaj
Azure Sev1 "API-Availability-Down" 54 CLOSE-STALE live curl 200 na sva tri qody.ba domena, zadnja pojava 07-21
Azure Sev1 "5xx-alert" 33 CLOSE-STALE live curl 200 bilko.cloud, zadnja pojava 07-17
Azure budget threshold exceeded 2 KEEP-OPEN alai-monthly-budget: amount=5500, currentSpend=15292 — jedina od 284 stavki gdje je uslov u trenutku triage-a i dalje VERIFIED true
Action-group email verification 1 UNVERIFIED CLI ne izlaže status verifikacije, nije pretpostavljeno
CEO mail 5 KEEP-OPEN (politika, nikad auto-close) —
Ermin Zatega / Avaz 2 KEEP-OPEN-PARKED CEO nalog, ne djeluj bez nove prijave
Test sender / SENTINEL health 3 CLOSE-STALE verifikovano živo (launchctl exit=0)
Ostalo (self-notes + vanjska korespondencija) 35 UNVERIFIED zaseban mail-triage pass, van scope-a ovog MC taska

Evidence: ~/system/evidence/106285/triage-2026-07-24.md (puni breakdown), subtask1-acceptance-replay.txt, enumeration-284-full.csv, non-ci-tasks.csv, Bilko-builds-full.json, QODY-builds-full.json, bilko-approvals.json, qody-approvals.json.

Bitna napomena o pokrivenosti fixa (nije bilo dio brief-a, otkriveno pri pisanju ove stranice): isNewsletterOrNotification() CI grana pokriva SAMO azuredevops@microsoft.com / azure-pipelines@ / noreply@dev.azure.com (Azure DevOps CI notifikacije = 149 od 239). azure-noreply@microsoft.com (Azure Monitor Sev1/budget alerti = 90 od 239) NIJE pokriven ovim guard-om — grep na inbox-watcher.js ne nalazi taj sender pattern nigdje u fajlu. Ovih 90 zatvoreno je samo kao jednokratni bulk-close u triage-u; ako se Sev1/budget alert ponovo desi, novi task će opet biti kreiran na H prioritetu. Ovo nije bug u smislu netačnosti fixa (fix radi tačno ono što je opisan da radi za CI build notifikacije) — ali je gap u pokrivenosti koji vrijedi zatvoriti prije nego se flood ponovi.

6. Šta ostaje otvoreno (pošteno, bez uljepšavanja)

Reference

ALAI Backup Strategy

ALAI Backup Strategy

⚠️ STALE-DOC BANNER (2026-08-08, MC #106946): the Layer 3 block below (SSH to Azure VM vm-alai-lightrag, 4 bare volume names) is stale — since the 2026-08-03 migration (#106747) the backup runs entirely local on this Mac against 7 lr106747-*-prefixed volumes, no SSH. Corrected, current detail: LightRAG Backup runbook (updated 2026-08-08). Not fully rewriting this page — see MC #106946 evidence for what changed and why.

Status: LIVE — verified against running processes, crontab, launchctl and logs on 2026-07-28. Owner: FlowForge (infra) / John (orchestration) Supersedes for accuracy: ~/system/architecture/backup-strategy-2026-04-20.md (that document is a PLAN dated 2026-04-20; several of its proposals were never activated or were later reversed by CEO decision — see §5 "Divergence from the original plan"). This page reflects what is actually running today.

1. Architecture — 4 layers, verified live

ANVIL (Mac Studio, 100.103.49.98)
│
├── Layer 1 — Git (hourly)
│     tools/hourly-backup.sh — cron 0 * * * *
│     Commits + pushes ~/system, ~/ALAI, ~/.claude, and every repo under ~/projects/*/*
│     to its GitHub/origin remote. Silent-fail-safe: logs push failures, retries next hour.
│
├── Layer 2 — SQLite (continuous + daily + 2x offsite)
│     PRIMARY: litestream replicate daemon (LaunchAgent com.alai.litestream, PID confirmed
│       live 2026-07-28, running since 2026-07-14). Config: ~/system/config/litestream.yml.
│       64 databases, WAL-shipping, sync interval 1s–300s depending on P0/P1/P2 tier.
│       ALL 64 replicas are `type: file` (LOCAL) → ~/backups/litestream/<db>/ (1.1 GB local,
│       verified 2026-07-28). ZERO Azure (`type: abs`) replicas remain — see §5.
│     SECONDARY (local snapshot): tools/db-backup.sh — cron 0 3 * * *, 5-day local retention,
│       ~/system/backups/databases/
│     OFFSITE #1: tools/rclone-backup.sh — launchd daily 03:00 (com.john.rclone-backup) →
│       Backblaze B2 (b2-alai:alai-studio-backup/system-databases/ + /claude-memory/).
│       Confirmed live run 2026-07-28T03:00Z, OK.
│     OFFSITE #2: daemons/offsite-backup.sh — launchd every 6h (com.john.offsite-backup) →
│       Backblaze B2 (b2-alai:alai-studio-backup/databases/, /config/, /rules/, /specs/,
│       /tools/, /claude-hooks/, /claude-agents/, /claude-skills/, /claude/).
│       Confirmed live run 2026-07-28T03:36Z, 9/9 targets OK, 85s.
│
├── Layer 3 — LightRAG / Neo4j Docker volumes
│     tools/lightrag-backup.sh — launchd weekly Sunday 04:00 (com.alai.lightrag-backup)
│     SSH to Azure VM vm-alai-lightrag (20.240.61.67) → docker compose stop → tar each of
│       4 volumes (lightrag-data, lightrag-kg, lightrag-cache, lightrag-neo4j-data) →
│       docker compose start → SHA-256 manifest.
│     SCP snapshot back to Mac Studio (~/system/backups/lightrag/, local safety net, keep 4).
│     Azure offsite upload → Storage Account `plockfrontstaging` (swedencentral),
│       Container `lightrag-backup`, Cool tier, keep last 8 snapshots (~8 weeks).
│     Full detail: [LightRAG Backup runbook](./lightrag-backup.md)
│
└── Layer 4 — Full ANVIL loss
      No fully-automated one-shot bootstrap is active today. Recovery = re-run Layers 1-3
      restore procedures on a freshly provisioned Mac. See Disaster Recovery Runbook.

2. Azure storage inventory (tool-verified 2026-07-28)

Account Region Purpose Status
alaibackups0ebb (RG alai-backups-rg) — Originally scoped for SQLite (system-db-backups), git bundles (system-git-bundles), Bitwarden exports (bitwarden-exports) per the April plan Scaffolded, not the active path. LaunchAgent com.alai.azure-db-backup.plist.disabled has been disabled since creation (file dated 2026-04-20). Litestream did target this account before the 2026-07-13 cost decision moved all 64 DB replicas to local file replicas (see §5). Current default az-cli SP identity gets AuthorizationFailed reading this account/RG — could not independently confirm current RBAC state; do not assume access without re-verifying.
plockfrontstaging (swedencentral) Hot LightRAG/Neo4j weekly offsite (container lightrag-backup, Cool tier per-blob) Active — this is a pre-existing Plock storage account being reused for LightRAG backup, not a dedicated ALAI backup account.
Backblaze B2 (b2-alai:alai-studio-backup, non-Azure) — SQLite snapshots, config/rules/specs/tools, .claude memory/hooks/agents/skills Active, two independent daemons (rclone-backup.sh daily, offsite-backup.sh 6-hourly).

3. Retention policy (as configured, verified)

Layer Mechanism Retention
Git GitHub remote history Unlimited (git history)
SQLite — litestream local ~/backups/litestream/<db>/ 24h–168h per DB tier (P0 financial longest)
SQLite — db-backup.sh local ~/system/backups/databases/ 5 days
SQLite — B2 offsite rclone sync (mirrors latest snapshot) Governed by B2 bucket lifecycle (not independently re-verified this session)
LightRAG/Neo4j — local ~/system/backups/lightrag/ Last 4 snapshots
LightRAG/Neo4j — Azure (plockfrontstaging) Cool tier blob Last 8 snapshots (~8 weeks)

The April plan's proposed 30-day-Cool → Archive → 365-day-delete lifecycle on alaibackups0ebb applies to the SQLite/git-bundle layer that was never activated in production — treat that lifecycle description as design intent, not live policy.

4. Cost (verified against live config, not the April estimate)

5. Divergence from the original 2026-04-20 plan — why, and the authority for it

CEO decision, verbatim, 2026-07-13 (MC #105462): "Prebaci lokalno! Nemamo placene korisnika kad to dodje ide backup na azure do tada save money gdje mozemo." ("Move it local! We don't have paying customers — when that happens, backup goes to Azure, until then save money wherever we can.")

Context: MC #105462 — litestream's per-sync-tick ListBlobs full-prefix enumeration against alaibackups0ebb (two hot telemetry DBs, flywheel/mission-control, at ~113K/~102K blobs each) was driving up to 518 GB/day list-egress (~370 NOK/day). After phased mitigation attempts (sync-interval tuning, lifecycle rules) proved insufficient, the CEO's second, later, better-informed decision (received ~10 hours after an earlier, more conservative one — the later decision took precedence per session record) was to migrate all litestream replicas to local file targets rather than keep paying Azure list-egress with no paying customers yet. Full incident record: Azure litestream egress saga + backup policy — MC #105462.

Practical effect: the April plan's Layer 2 (SQLite → Azure Blob, dedicated alaibackups0ebb account) is stood up in config/scripts but intentionally not the live path. B2 (Backblaze) offsite + local litestream replicas carry that layer today. LightRAG/Neo4j (Layer 3) was unaffected — it is a low-frequency (weekly), low-blob-count workload on a different storage account, so it kept its original Azure design.

IAM note: the SP configured for the Azure Blob SQLite path (alai-backup-writer / appid 1a0b3018-…, per ~/system/config/litestream.yml and azure-backup.env) is the same appid used as the general-purpose az-cli identity elsewhere in the system. VAULT_EXPORTER_APPID in azure-backup.env is still FILL_AFTER_SP_CREATION — that narrower-scoped SP for Bitwarden exports was never created. Do not assume it exists.

6. Reactivation trigger

Per the CEO decision above, the Azure Blob SQLite backup path (alaibackups0ebb) should be reactivated when ALAI has paying customers. Reactivation requires, at minimum: (1) re-verify RBAC on alai-backups-rg/alaibackups0ebb (current az-cli SP identity failed this check on 2026-07-28), (2) re-enable com.alai.azure-db-backup.plist (currently .disabled), (3) re-point litestream replicas for at least the P0-financial tier back to type: abs, (4) re-run the lifecycle-policy fix from #105490 (an account-wide lifecycle rule without prefixMatch archived 35 dormant DBs' L0 blobs during the original incident — must be scoped before reactivation).

Document Owner: Lexicon (Skillforge role) Last Verified: 2026-07-28 — process list (litestream replicate, live PID), crontab -l, launchctl list, LaunchAgent plists, live log tails (offsite-backup.log, rclone-backup-20260728.log), az account show / az storage account show (AuthorizationFailed, documented as-is), and ~/system/config/litestream.yml (64/64 DBs confirmed type: file).

Disaster Recovery Runbook

Disaster Recovery Runbook

Status: LIVE — restore commands below verified against the config/scripts actually running on 2026-07-28. See ALAI Backup Strategy for the architecture these steps restore from.

IMPORTANT — read before following any restore procedure: ~/system/architecture/litestream-restore-runbook.md (dated 2026-04-20) documents litestream restore ... abs://alaibackups0ebb/... commands. Those commands target the Azure Blob replica path, which is no longer being written to — as of the CEO cost decision on 2026-07-13 (MC #105462), all 64 litestream-managed databases replicate to local file replicas only (~/backups/litestream/<db>/), confirmed live in ~/system/config/litestream.yml on 2026-07-28. Any data in alaibackups0ebb predates 2026-07-13 and will be stale. Use the file-replica commands in Layer 2 below, not the old abs:// commands, unless the Azure Blob SQLite path has since been reactivated (check §6 of the Backup Strategy page first).

Layer 1 — Git (fastest, use first for any code/config loss)

Source of truth: GitHub/origin remote for each repo (~/system, ~/ALAI, ~/.claude, ~/projects/*/*).

# Clone fresh
git clone <remote-url> <target-dir>

# Or, if the local repo exists but is behind/corrupted:
cd <repo>
git fetch origin
git reset --hard origin/<branch>    # DESTRUCTIVE — confirm no uncommitted work worth keeping first

Recovery point objective (RPO): up to 1 hour of uncommitted work (hourly-backup.sh cron interval, 0 * * * *). Check ~/system/logs/hourly-backup-cron.log for the last successful push timestamp before relying on this.

Layer 2 — SQLite databases

2a. Normal case — restore from local litestream replica (PRIMARY, live today)

# Example: mission-control.db
litestream restore \
  -o /tmp/mission-control-restored.db \
  ~/backups/litestream/mission-control

sqlite3 /tmp/mission-control-restored.db "PRAGMA integrity_check;"
sqlite3 /tmp/mission-control-restored.db "SELECT COUNT(*) FROM tasks;"

# If good, replace the live DB (stop anything writing to it first):
cp /tmp/mission-control-restored.db ~/system/databases/mission-control.db

Repeat per-database using the replica path from ~/system/config/litestream.yml (replicas[].path, all currently under ~/backups/litestream/<db>/). Lag from primary under normal operation: sub-minute (1s–300s depending on DB tier — P0-critical/financial sync fastest).

2b. ANVIL local disk lost entirely — restore from Backblaze B2 offsite

# List what's there
rclone lsd b2-alai:alai-studio-backup/system-databases/

# Pull a specific DB snapshot down
rclone copy b2-alai:alai-studio-backup/system-databases/mission-control.db /tmp/restore/

# Or pull the full daily snapshot set
rclone copy b2-alai:alai-studio-backup/databases/ /tmp/restore-daily/

Two independent B2 jobs exist — rclone-backup.sh (daily 03:00, system-databases/ + claude-memory/) and offsite-backup.sh (every 6h, databases/ + config/ + rules/ + specs/ + tools/ + .claude/*). Check both prefixes; freshness may differ by up to 6h between them.

Caveat: B2 snapshots are point-in-time copies of the litestream local replica at sync time, not continuously replicated — worst-case data loss window is the sync interval of whichever job ran last (up to 6h), not the sub-minute litestream RPO.

2c. If the Azure Blob SQLite path has been reactivated since 2026-07-28

Only if ~/system/config/litestream.yml shows type: abs replicas again (verify before using):

export AZURE_CLIENT_ID="1a0b3018-0c31-474b-918f-531b0a29a669"
export AZURE_CLIENT_SECRET="<retrieve from Bitwarden: alai-backup-writer>"
export AZURE_TENANT_ID="3454a03f-20b4-4bda-a116-2293c459aecd"

litestream restore \
  -o /tmp/mission-control-restored.db \
  abs://alaibackups0ebb/system-db-backups/litestream/mission-control

Full detail (promotion-to-write-primary on a replacement host, multi-scenario): ~/system/architecture/litestream-restore-runbook.md — usable once the abs:// path is confirmed live again; do not follow it blindly today.

Layer 3 — LightRAG / Neo4j Docker volumes

Source of truth (corrected 2026-08-08, MC #106946): this Mac (Makinja-sin-Mac-Studio), not the Azure VM — live since the 2026-08-03 migration (#106747). Backup script runs entirely local docker run/docker compose, no SSH involved.

Full procedure with all 3 restore scenarios (this machine, from Azure Blob offsite download, throwaway-volume verification) is maintained in LightRAG Backup runbook — do not duplicate here, it is live and current. Summary:

# 1. Pick snapshot (local safety net or download from Azure Blob first)
SNAPSHOT=~/system/backups/lightrag/<timestamp>
cd "$SNAPSHOT" && shasum -a 256 -c MANIFEST.sha256

# 2. Restore all 7 volumes (post-migration live names, lr106747-* prefix — MC #106946)
for vol in lr106747-data lr106747-kg lr106747-cache lr106747-neo4j-data lr106747-neo4j-import lr106747-neo4j-logs lr106747-neo4j-plugins; do
  docker volume rm $vol || true
  docker volume create $vol
  docker run --rm -v $vol:/dst -v "$SNAPSHOT":/src alpine tar xzf /src/${vol}.tar.gz -C /dst
done

cd ~/system/lightrag-local && docker compose -f docker-compose.local.yml up -d
curl http://localhost:9621/health   # expect {"status":"healthy"}

To pull from the Azure offsite copy instead of the local safety net:

source ~/system/config/azure-lightrag-backup.env
az storage blob download-batch \
  --account-name $AZURE_STORAGE_ACCOUNT --account-key "$AZURE_STORAGE_KEY" \
  --source $AZURE_STORAGE_CONTAINER --destination ~/system/backups/lightrag/azure-restore-<TS> \
  --pattern "<TS>/*"

Layer 4 — Full ANVIL loss (new Mac, everything gone)

No single automated bootstrap script covers this end-to-end today; ~/system/scripts/azure-blob-bootstrap.sh and ~/system/scripts/migrate-lightrag-to-azure.sh exist but were written against the April-plan Azure Blob SQLite path, which is not the live path (see Layer 2 note above) — treat them as reference, not a turnkey script, until re-validated.

Manual sequence:

  1. Provision a new Mac (or VM), install prerequisites: brew install git sqlite3 rclone litestream docker.
  2. Layer 1 — Git: clone ~/system, ~/ALAI, ~/.claude, and every ~/projects/*/* repo from their GitHub remotes.
  3. Layer 2 — SQLite: restore each database from Backblaze B2 (§2b — this is the only surviving copy if ANVIL's local disk, including the litestream local replicas under ~/backups/litestream/, is gone).
  4. Layer 3 — LightRAG/Neo4j: the Azure VM (vm-alai-lightrag, 20.240.61.67) is a separate host from ANVIL — if only ANVIL is lost, this layer is untouched. If the Azure VM is also lost, restore from plockfrontstaging/lightrag-backup per §Layer 3 above onto a freshly provisioned VM.
  5. Reconfigure launchd: re-load LaunchAgents for com.alai.litestream, com.john.rclone-backup, com.john.offsite-backup, com.alai.lightrag-backup (all found under ~/Library/LaunchAgents/ in the repo you just restored — none are currently .disabled except the Azure-Blob-SQLite ones, which stay disabled per the CEO cost decision unless explicitly reactivated).
  6. Verify: crontab -l shows the 3 cron jobs (gotcha-health, db-backup, hourly-backup); launchctl list | grep -E "litestream|backup" shows all 4 active LaunchAgents loaded; curl http://localhost:9621/health (or the Azure VM equivalent) returns healthy.

This layer has not been drill-tested end-to-end — no MC evidence of a live full-ANVIL-loss rehearsal was found this session. Flagging per ZAKON PLAN: a Proveo/Angie Jones validation task simulating this (stop services, wipe a scratch volume, restore, verify) should exist before this runbook is trusted under real incident pressure.

Document Owner: Lexicon (Skillforge role) Last Verified: 2026-07-28 — cross-checked against ~/system/config/litestream.yml (live, 64/64 type: file), live litestream replicate process, crontab -l, launchctl list, and ~/system/docs/runbooks/lightrag-backup.md.

Edita PA Runbook

Edita PA Runbook

TLDR

Identity

Field Value
Name Edita
Role Personal Assistant, ALAI Holding AS
Boss Alem Basic (CEO, alem@alai.no)
Manager John (AI Director)
Primary mailbox info@alai.no (Migadu)
System prompt ~/system/agents/pa/edita-system-prompt.md

Language: mirrors the user (Bosnian/English/Norwegian mix OK). Tone: direct, warm, professional, max 3 sentences for routine replies. Edita never says "I can't" — she escalates to John instead.

What Edita does

What Edita does NOT do

Escalation rules (always → John → Alem)

Architecture — the loop

Script: ~/system/agents/pa/edita-loop.js (extends daemon-base.js — signal handling, heartbeat, circuit breaker).

masterTick()
  A. emailTriageTick()          — every 5 min   [ACTIVE]
  B. calendarTick()             — every 15 min  [STUB — Faza 3, not active]
  C. memoryConsolidationTick()  — every 30 min  [STUB — Faza 3, delegated to edita-memory-writer.js]
  D. incomingMessageHandler()   — event-driven  [ACTIVE — Slack/Telegram, Faza 2]
  validationTick()              — every 6h, always active (incl. dry-run)

Email triage flow (emailTriageTick):

  1. Pull unclassified mail for info@alai.no from ~/system/databases/email-inbox.db.
  2. Classify via Ollama (TRIAGE_MODEL = llama3.1:8b, OLLAMA_HOST default http://localhost:11434).
  3. Write-back classification even in dry-run, so items aren't re-examined every tick.
  4. If reply warranted: build draft, write to ~/system/intake/drafts/draft-<ts>-<subject>.md.
  5. Gate 3 check: autoReplyEnabled = config.auto_reply_enabled || AUTO_REPLY_ENABLED (AUTO_REPLY_ENABLED is hardcoded false in code) — if false, hold and log GATE3_HOLD, notify John. No send happens.

Key config/paths

Const Path
INTAKE_CONFIG ~/system/config/intake-config.json
DRAFTS_DIR ~/system/intake/drafts/
DECISIONS_LOG ~/system/logs/edita-decisions.jsonl
HEARTBEAT_FILE ~/system/logs/edita-loop-heartbeat.json
SYSTEM_PROMPT_FILE ~/system/agents/pa/edita-system-prompt.md
EMAIL_INBOX_DB ~/system/databases/email-inbox.db
VIP_CONTACTS ~/system/agents/pa/memory/contacts-vip.json

Run modes

node ~/system/agents/pa/edita-loop.js               # live (LaunchAgent default)
node ~/system/agents/pa/edita-loop.js --dry-run      # Faza 1: logs decisions only, no drafts sent
node ~/system/agents/pa/edita-loop.js --allow-archive  # next safe step after dry-run: allows ARCHIVE action, still no sends

Operations

# Status
launchctl list | grep com.john.edita-loop
cat ~/system/logs/edita-loop-heartbeat.json

# Restart
launchctl kickstart -k gui/$(id -u)/com.john.edita-loop

# Recent decisions
tail -50 ~/system/logs/edita-decisions.jsonl

# Pending drafts (never auto-sent — review before send)
ls -la ~/system/intake/drafts/

Plist: ~/Library/LaunchAgents/com.john.edita-loop.plist.

Troubleshooting

Edita not triaging new mail

  1. Confirm daemon is running: launchctl list | grep com.john.edita-loop (should show a PID, not -).
  2. Check heartbeat freshness: cat ~/system/logs/edita-loop-heartbeat.json — stale timestamp means the tick loop died silently; kickstart it.
  3. Check Ollama circuit breaker: grep "circuit breaker open" ~/system/logs/edita-loop.log — if tripped, Ollama (localhost:11434) is unreachable; test with curl -s localhost:11434/api/tags.
  4. Confirm info@alai.no is actually being scanned — see [[Email System Runbook]] for the account/daemon relationship (separate daemon, email-agent.js, ingests mail into email-inbox.db; Edita reads from that same DB, it does not do its own IMAP fetch).

A draft never turned into a sent email This is expected — AUTO_REPLY_ENABLED = false is a deliberate CEO gate, not a bug. Drafts sit in ~/system/intake/drafts/; John or Alem must review and send manually via the paths in [[Email System Runbook]].

Calendar reminders / auto-memory-consolidation not happening Expected — both are Faza 3 stub code in edita-loop.js (calendarTick, memoryConsolidationTick are no-ops that just log a debug line). Not a regression; they were never activated.

Hosting Migration Log — basicconsulting.no + bilko.io (MC #8523)

Hosting Migration Log — basicconsulting.no + bilko.io

Author: Skillforge (Lexicon) · Date: 2026-07-28 · Trigger: MC #8523 (BookStack documentation task)

This page documents the tool-verified hosting history and current live state of two domains: basicconsulting.no (ALAI/Basic Consulting corporate site) and bilko.io (Bilko Serbian-market landing page). Source facts below are drawn from email #2072 (2026-04-21 hosting triage), the ALAI Static Hosting Blueprint (MC #8481), the Bilko CSP Cleanup doc (MC #10440), GitHub Actions deploy history, and live dig/curl checks run 2026-07-28.

1. Starting point — 2026-04-19/21 triage (email #2072)

FlowForge audit (2026-04-19) found ALAI hosting spread across 5 platforms, 12 live domains, 22 Vercel projects (mostly orphaned):

Platform Held Status (2026-04-21)
Cloudflare Pages alai.no, basicconsulting.no New standard per blueprint MC #8481
Azure VM (4.223.110.181) BookStack, Vaultwarden, Documenso, Planka, Grafana, app.getdrop.no Internal tools + Drop prod
GCP Cloud Run bilko-api, bilko-web, bilko-intesa-demo Bilko dynamic backend
Vercel basicfakta.no + 21 stale/orphan projects Cleanup needed
GitHub Pages snowit.ba Client project

bilko.io at this point: unreachable (timeout) — domain purchased (One.com, per 2026-02-19 session decision) but no hosting attached.

2. Static migration to Cloudflare Pages (MC #8481)

Per the ALAI Static Hosting Blueprint (MC #8481, 2026-04-20):

Cloudflare Pages was selected over GCP/AWS/Azure Static Web Apps on cost (€0 vs €12-14/mo for 12 sites), native git-push deploys, unlimited custom domains/free SSL, and alignment with ALAI's existing Cloudflare DNS footprint.

3. bilko.io — from unreachable to live static landing

4. basicconsulting.no — retired as Bilko dev host, still live as corporate site

Two distinct threads involve basicconsulting.no and must not be conflated:

a) Corporate site (this doc's primary subject): basicconsulting.no itself remains live on Cloudflare Pages (migrated 2026-04-20, section 2 above). Verified 2026-07-28: resolves via Cloudflare (104.21.35.64 / 172.67.214.88), HTTP/2 200, serving "Basic Consulting | Premium IT-rådgivning" (Norwegian corporate landing page).

b) Bilko dev-config subdomains (retired): Per MC #10440 (2026-07-01), stale basicconsulting.no subdomain entries (bilko-demo.basicconsulting.no, bilko-demo-api.basicconsulting.no, and a *.basicconsulting.no wildcard) were removed from Bilko's apps/web/next.config.js dev CSP and allowedDevOrigins, following the app's demo infrastructure migration to alai.no hosts. curl verification at the time: bilko-demo.basicconsulting.no — TLS handshake completes, no HTTP response; bilko-demo-api.basicconsulting.no — HTTP 502 (Cloudflare proxy up, origin dead). Verified again 2026-07-28: bilko-demo.basicconsulting.no resolves to a mixed CNAME/A-record set pointing at an orphaned GCP Cloud Run host (bilko-web-*.a.run.app + Google IP range) — consistent with the MC #10440 finding of a dead legacy origin. These subdomains are not part of the live corporate site and are not expected to resolve to working content.

4a. Cross-reference — related existing BookStack page (discrepancy flagged, not resolved here)

An existing page, "ALAI Domain Migration — basicconsulting.no → alai.no" (BookStack page id 2666, book Operations, created 2026-04-19, last updated 2026-07-05, tagged staleness: needs-review), documents the basicconsulting.no Cloudflare zone (4670dbd0acfeab4174ac0d4746d11ea0) at the DNS/tunnel level: root + docs, sign, bilko-demo, www subdomains proxied via Cloudflare Tunnel to the Azure VM (4.223.110.181), separate from the CF-Pages-hosted apex corporate site covered in section 4a above.

Discrepancy found: that page (last updated 2026-07-05) lists bilko-demo.basicconsulting.no as an "Active service." This conflicts with MC #10440 (2026-07-01, 4 days earlier) which found the same host dead by curl (TLS handshake completes, no response), and with this doc's own 2026-07-28 re-verification (orphaned GCP Cloud Run origin). The Operations-book page is already self-flagged needs-review — this log does not attempt to resolve the discrepancy, only surfaces it so a future reader does not treat the older page as current truth over the two independent 2026-07 dead-host findings.

5. Current live state summary (verified 2026-07-28)

Domain Platform HTTP status Serves
basicconsulting.no Cloudflare Pages 200 ALAI/Basic Consulting corporate landing (Norwegian)
bilko.io Cloudflare Pages 200 Bilko Serbia static landing (sr-Latn), part of bilko.cloud/bilko.rs/bilko.company hreflang network
bilko-demo.basicconsulting.no dead (orphaned GCP Cloud Run origin) n/a Not live — retired per MC #10440

6. Open items (not resolved by this log; flagged for follow-up, not actioned here)

Sources: email #2072 (message id in ~/system/databases/email-inbox.db, row 2072), ~/system/specs/ALAI-STATIC-HOSTING-BLUEPRINT.md (MC #8481), ~/system/docs/reconciliation/bilko-mc-10440-csp-cleanup.md (MC #10440), GitHub Actions notification history in email-inbox.db, live dig/curl checks 2026-07-28.

Network Watchdog Response Procedures

Network Watchdog Response Procedures

Status: Active runbook
Created: 2026-07-28
Owner: FlowForge / John
Primary design link: ~/system/architecture/network-watchdog-design.md
Source files verified before writing:

Purpose

Use this runbook when Network Watchdog emits a network, DNS, target-health, or multi-target alert. The goal is to classify the alert, confirm whether it is real or known-noisy, restore the affected target, and escalate before dependent services fail.

Implementation inventory

There are two Network Watchdog implementations in the system tree. Verify which one is active before acting:

  1. Detailed shell checker: ~/system/tools/network-watchdog.sh
    • Checks: Internet, FORGE LAN, Tailscale mesh, Azure Vault, BookStack, DNS MX, DNS drift.
    • State: /tmp/network-watchdog-fails/<target>.
    • Log: ~/system/logs/network-watchdog.log.
    • Design reference: ~/system/architecture/network-watchdog-design.md.
  2. KeepAlive JS daemon: ~/system/daemons/network-watchdog.js
    • Checks: default gateway ping, DNS resolution, Internet connectivity.
    • Heartbeat: ~/system/logs/network-watchdog-heartbeat.json.
    • LaunchAgent file in repo: ~/system/daemons/launchagents/com.john.network-watchdog.plist.

Important: the design document names com.alai.network-watchdog, while current daemon registry files reference com.john.network-watchdog. Do not reload or edit LaunchAgents during an incident until the current label/path is verified on the machine.

Alert interpretation

Shell checker levels

Level Meaning Default threshold Response
WARN First failed check; may self-recover 1 failed run Confirm with one manual probe, watch next cycle.
ALARM Sustained failure Usually 3 consecutive failed runs, about 15 minutes Start triage and recovery steps.
PANIC Critical sustained or broad outage 5+ consecutive failures, Internet 2+, or 3+ targets down Treat as active infrastructure incident. Escalate immediately.

Special cases from verified files:

JS daemon levels

~/system/daemons/network-watchdog.js alerts after 3 consecutive failures for gateway, DNS, or Internet checks. It has a 10-minute alert cooldown and sends to Slack plus macOS notification when those channels work.

First triage steps for any alert

  1. Preserve evidence first. Do not restart services before capturing alert text and recent logs.
  2. Identify implementation and target. Determine whether the alert came from shell target names (forge-lan, tailscale-mesh, azure-vault, azure-docs, dns-mx, dns-drift, internet) or JS target names (gateway, dns, internet).
  3. Read recent logs.
tail -120 ~/system/logs/network-watchdog.log
  1. Check current daemon state before touching it.
launchctl list | grep -i network-watchdog || true
launchctl print gui/$(id -u)/com.john.network-watchdog
  1. Check counters for shell checker alerts.
ls -la /tmp/network-watchdog-fails
for f in /tmp/network-watchdog-fails/*; do [ -f "$f" ] && printf '%s=' "$(basename "$f")" && cat "$f"; done
  1. Classify scope.
    • One target only: likely target-specific.
    • DNS MX or DNS drift only: likely DNS/provider/config issue.
    • Internet + several targets: likely local network or host outage.
    • Tailscale only with high fail count: check whether it is the known steady-state offline-node pattern before paging.

Target-specific recovery procedures

1. Internet connectivity (internet)

Alert signals: shell Internet unreachable, JS internet failure, PANIC after 2+ shell failures.

Confirm:

ping -c 2 1.1.1.1
curl -sf --max-time 5 https://1.1.1.1/cdn-cgi/trace
route -n get default

Recover:

  1. If default route is missing, inspect the active network interface before changing anything.
  2. If Wi-Fi/Ethernet is down, restore local connectivity from macOS Network settings or the physical network path.
  3. Re-run the confirmation commands.
  4. If Internet is down and 3+ watchdog targets also fail, escalate as systemic outage.

2. Gateway (gateway, JS daemon)

Alert signals: JS daemon reports GATEWAY FAILURE and includes the detected gateway IP.

Confirm:

route -n get default
ping -c 3 <gateway-ip-from-alert>

Recover:

  1. Verify the gateway IP in the alert matches route -n get default.
  2. If gateway ping fails but Internet works, treat as gateway ICMP filtering/noise and monitor.
  3. If gateway and Internet both fail, recover local network path first.
  4. Escalate if local network cannot be restored from the host.

3. DNS resolution (dns, dns-mx, dns-drift)

Alert signals: JS DNS FAILURE, shell MX records unexpected, or shell DNS drift detected.

Confirm:

dig +short +time=3 google.com
dig +short MX alai.no @1.1.1.1
dig +short A alai.no @1.1.1.1
dig +short A alai.no @8.8.8.8

Recover DNS resolution failure:

  1. If all DNS queries fail, confirm Internet first.
  2. If Internet works but DNS fails, switch/test resolver path before changing app services.
  3. Re-run dig +short +time=3 google.com and wait one watchdog cycle.

Recover MX failure:

  1. Current shell script expects migadu.com in dig +short MX alai.no @1.1.1.1.
  2. If Migadu records are missing, treat as mail-delivery risk.
  3. Check Cloudflare DNS for alai.no and restore Migadu MX records.
  4. Re-run the MX check against 1.1.1.1 and 8.8.8.8.

Handle DNS drift:

4. FORGE LAN (forge-lan)

Alert signals: shell FORGE (10.0.0.2) unreachable.

Confirm:

ping -c 3 -W 2000 10.0.0.2

Recover:

  1. Confirm whether FORGE is expected to be powered on and on the LAN/Thunderbolt path.
  2. Check physical link, power state, and host reachability through any secondary access path available at the time.
  3. If FORGE is intentionally offline, document the maintenance window and suppress downstream work that depends on it.
  4. If FORGE is unexpectedly down for ALARM/PANIC thresholds, escalate to infrastructure owner for hands-on host recovery.

5. Tailscale mesh (tailscale-mesh)

Alert signals: shell reports offline node count and names.

Confirm:

tailscale status

Recover:

  1. Identify whether the offline nodes are expected idle/offline devices or required infrastructure hosts.
  2. If the alert contains the known historical pattern makinja-sin-mac-studio, basicass-mac-mini, iphone181 with a very high fail count, treat as steady-state unless current work depends on those nodes.
  3. For required nodes, check Tailscale service on the affected node and re-auth/reconnect only if you have current host access.
  4. Re-run tailscale status and watch the next watchdog cycle.

6. Azure Vault (azure-vault)

Alert signals: shell Azure Vault unhealthy, expected HTTP 200/302.

Confirm:

curl -s -o /dev/null -w '%{http_code}\n' --max-time 10 https://vault.alai.no/healthz

Recover:

  1. If HTTP is 200/302, reset/observe; it was transient.
  2. If HTTP is 5xx/timeout, check whether Internet and DNS are healthy first.
  3. If only Vault is unhealthy, follow the Vault service runbook/host access path and capture HTTP status plus timestamp.
  4. Escalate as secrets-access incident if deployments, agents, or BookStack sync are blocked by Vault unavailability.

7. BookStack/docs (azure-docs / BookStack)

Alert signals: shell BookStack unhealthy, expected HTTP 200/302 for https://docs.alai.no.

Confirm:

curl -s -o /dev/null -w '%{http_code}\n' --max-time 10 https://docs.alai.no

Recover:

  1. If HTTP is 200/302, mark transient and watch next cycle.
  2. If public docs are down but local BookStack is available, use ~/system/context/docs/runbooks/bookstack.md for container/API/database recovery.
  3. If BookStack API is rate-limited (429 Too Many Attempts), stop automation retries and wait for the rate window before sync attempts.
  4. If docs are inaccessible during an active incident, preserve this local runbook path: ~/system/docs/runbooks/network-watchdog-response-procedures.md.

Escalation paths

Escalate based on scope and business impact:

  1. WARN single target: John/FlowForge watches logs; no CEO interruption unless target blocks current work.
  2. ALARM single target: John/FlowForge begins recovery. Escalate to hands-on host owner if physical access is needed.
  3. PANIC or 3+ targets down: Treat as systemic incident. Notify CEO with one factual line: target count, failed target names, first-failure time, and current action.
  4. Mail/DNS MX broken: Escalate as mail-delivery risk after confirming Migadu MX records are missing from public resolvers.
  5. Secret access blocked: Escalate if Vault outage blocks deploys, agents, or credential retrieval.
  6. Alert flood: Do not add more alerts. Apply cooldown/suppression logic first and use the 2026-05-15 incident pattern below.

Historical incident patterns

Verified from design, incident, and MC snapshot files:

  1. 2026-04-19 to 2026-04-20 network incident cluster (network-watchdog-design.md): ANVIL OOM with network aspects, alai.no MX tampering through Cloudflare, FORGE 10.0.0.2 unreachable from 16:40-17:30, and ANVIL ping disabled before memory fix.
  2. Initial watchdog run 2026-04-20 17:43 (network-watchdog-design.md): FORGE was unreachable and Tailscale had 3 offline nodes; Internet, Azure Vault, BookStack, DNS MX, and DNS resolver consistency were healthy at that moment.
  3. 2026-05-15 Slack flood (slack-flood-2026-05-15.md): root cause was network-watchdog with 3 permanent-fail checks and zero cooldown. Actions included daemon unloads and network-watchdog disablement decision path.
  4. MC #100764 fix snapshot (state/lightrag-ingest-mc/100764.md): updated MX baseline to Migadu, added per-check 6-hour cooldown files under /tmp/network-watchdog-lastalert-<check>.ts, suppressed steady-state Tailscale offline after 100 cycles, and reduced Slack rate from about 60/hour to 1-2/hour.
  5. DNS drift false-positive pattern (network-watchdog.sh): Cloudflare anycast can make 1.1.1.1 and 8.8.8.8 return different A records; the current script logs this and disables Slack alerting for that check.

Post-incident closeout

After recovery:

  1. Capture final evidence: alert text, recent network-watchdog.log lines, confirmation command outputs, and affected target names.
  2. Confirm next watchdog cycle no longer increments the target counter.
  3. If a baseline changed intentionally, update both:
    • ~/system/tools/network-watchdog.sh
    • this runbook and/or ~/system/architecture/network-watchdog-design.md
  4. If the incident caused user-facing impact, create/update the relevant MC task with evidence and BookStack link.

CF Access Service Token Rotation

CF Access Service Token Rotation (docs.alai.no / BookStack)

MC: #99235 (cleanup) — follow-up to Proveo audit #99027 (2026-05-05), deferred from kelsey-hightower subtask 3 on #99027.

Background

docs.alai.no (BookStack, Azure VM 4.223.110.181) sits behind Cloudflare Access. All programmatic API calls (BookStack REST API, sync tooling) need two credential pairs:

  1. BookStack API token — Authorization: Token <token_id>:<token_secret> (BookStack user token)
  2. CF Access service token — CF-Access-Client-Id + CF-Access-Client-Secret headers (bypasses the CF Access login redirect for non-interactive clients)

Both live in ~/system/config/bookstack.json (local cache) and, for the BookStack token, in Vaultwarden item "BookStack API" (fe418612-0d17-4e80-9b2e-39ef7c213b61).

Finding: CF_ACCESS_CLIENT_SECRET leaked in git history

~/system/config/bookstack.json was git-tracked with the CF Access client secret in plaintext from 2026-02-12 through ~1775 auto-backup commits (repo ~/system, private, external exposure minimal per CEO risk assessment 2026-05-05).

CEO decision (2026-05-05): document, do NOT rotate the token immediately.

Status as of 2026-07-29 (MC #99235 cleanup pass):

Rotation procedure (when CEO approves)

  1. Generate new service token in Cloudflare Zero Trust dashboard → Access → Service Auth → Service Tokens → create/rotate the token scoped to docs.alai.no (and any other *.alai.no host sharing the same Access policy).
  2. Store in Vaultwarden as a dedicated item (do not reuse an unrelated item like b42cb5c2). Recommended: new item "BookStack CF Access" with fields cf-access-client-id / cf-access-client-secret, matching the field-name pattern bookstack-sync.js already expects when a vault item does carry these fields (loadConfigFromVault() reads fields.cf_access_client_id / fields.cf_access_client_secret from the "BookStack API" item — either add the fields there, or update ~/system/tools/bookstack-sync.js:118-130 to also check the new item name).
  3. Update ~/system/config/bookstack.json with the new pair (file is now gitignored — editing it will NOT create a new leak).
  4. Revoke the old token in Cloudflare Zero Trust once the new one is verified working: curl -I -H "CF-Access-Client-Id: <new>" -H "CF-Access-Client-Secret: <new>" https://docs.alai.no/api/docs → expect 200.
  5. Verify sync still works: node ~/system/tools/bookstack-sync.js status should list pages without Vault unreachable or HTTP 401/403 errors.
  6. Update this runbook with the new Vaultwarden item ID once done.

Prevention (already applied, MC #99235)

url-linter.js — CEO-facing URL Verification Gate

url-linter.js — Usage Runbook

Owner: CodeCraft Created: 2026-04-22 Purpose: Prevent broken URLs from reaching CEO. Enforces ZAKON URL.

When to Use

Run pre-ceo-publish.sh BEFORE sending ANY of these to Alem:

Trigger rule: "Every URL in a CEO-facing document MUST pass url-linter.js before publish."

Quick Start

# Gate check before CEO delivery (REQUIRED)
bash ~/system/hooks/pre-ceo-publish.sh ~/system/specs/my-doc.md

# Direct linter run
node ~/system/tools/url-linter.js ~/system/specs/my-doc.md

# Scan entire specs directory
node ~/system/tools/url-linter.js ~/system/specs/

# JSON output (for scripts)
node ~/system/tools/url-linter.js ~/system/specs/my-doc.md --json

All Options

node url-linter.js <file-or-dir> [options]

Options:
  --fix              Replace redirected URLs with final URLs; remove NXDOMAIN URLs and append a FIXME flag
  --json             Output JSON report (machine-readable)
  --fail-on=<N>      Exit 1 if N or more broken URLs (default: 1)
  --concurrency=<N>  Max parallel checks (default: 5)
  --timeout=<ms>     Per-URL curl timeout in ms (default: 15000)
  --cache=<path>     SQLite cache path (default: ~/system/state/url-linter-cache.db)
  --no-cache         Skip cache, always re-check every URL
  --cache-ttl=<h>    Cache TTL in hours (default: 6)
  --quiet            Only show failures and summary

Exit Codes

Code Meaning
0 All URLs pass (or below fail-on threshold)
1 One or more broken URLs found (at or above fail-on threshold)
2 Usage error or fatal (file not found, etc.)

What Gets Checked

URL patterns extracted from markdown:

Test method: curl -sSIL --max-time <N> <url>

Result Classification
HTTP 2xx PASS
HTTP 3xx followed by 2xx PASS (shown as REDIRECT)
HTTP 4xx, 5xx FAIL
NXDOMAIN (cannot resolve) FAIL
Timeout FAIL
Connection refused FAIL

Cache

Results are cached in ~/system/state/url-linter-cache.db (SQLite).

PostToolUse Hook (automatic)

The hook ~/system/hooks/url-linter-gate.sh runs automatically after Write/Edit on CEO-facing files. It runs async (does not block the write) but prints a warning to stderr if broken URLs are found.

CEO-facing patterns that trigger automatic check:

Hook activation is present in ~/.claude/settings.json under PostToolUse for Write|Edit|MultiEdit. The canonical hook block is:

{
  "matcher": "Write|Edit|MultiEdit",
  "hooks": [
    {
      "type": "command",
      "command": "bash ~/system/hooks/url-linter-gate.sh",
      "timeout": 60000,
      "async": true
    }
  ]
}

Pre-CEO Publish Script

bash ~/system/hooks/pre-ceo-publish.sh <file>

This wraps url-linter.js with:

Make this the last step before any CEO delivery.

Retroactive Audit

To scan all existing specs for broken URLs:

node ~/system/tools/url-linter.js ~/system/specs/ --json > /tmp/url-audit.json

Or run the full retroactive audit:

bash ~/system/hooks/pre-ceo-publish.sh ~/system/specs/

Incident Context

Origin: 2026-04-24 HR-FISK incident. John published www.porezna-uprava.gov.hr/bi/Stranice/Popis-informacijskih-posrednika.aspx (NXDOMAIN) to CEO in a "VERIFIED" document. URL was copied from web-search output without curl verification.

Actual working URL: porezna-uprava.gov.hr/hr/popis-informacijskih-posrednici/8019 (HTTP 200).

Root cause: RC5 (Petter Graff audit) — citation propagation without re-verification. This tool closes that gap permanently.

ZAKON URL (exact text)

Every external URL and email domain published in a CEO-facing document must have a fresh HTTP verification within 10 minutes of document delivery. Verification method: curl -sI --max-time 10 returning 2xx or 3xx. NXDOMAIN, 4xx, or timeout = document blocked. No exceptions for "found in registry" or "from web search". Web search results are unverified input, not verified facts.

Resolver #106388 — persona-gate blocks 'compliance' tasks (MC #105398)

Resolver #106388 — pi-orchestrator persona-gate blocks "compliance" tasks (MC #105398)

Summary

MC #106388 ("[RESOLVER] task_failures: 18 tasks paused/blocked in last 6h") was reviewed by reproducing the resolver-daemon.js 6h-window query directly against mission-control.db. 17 matching rows found (title said 18 — off-by-one, likely a status change between the daemon run and this review; immaterial to the diagnosis).

Breakdown (17 tasks in the reported window)

Root cause (confirmed, pre-existing, already filed and stuck)

~/system/kernel/pi-orchestrator.js (persona-gate personaKeywords list, around line 5893) includes the bare word 'compliance' alongside genuine healthcare terms (hipaa, cqc, clinical, emar, safeguarding). Any task whose title/description contains compliance AND has no required_agent set gets hard-blocked with no auto-recovery path — mislabeled as needing a healthcare persona even when the compliance context is tax/GDPR/security/eRačun.

Suggested fix (not yet applied — kernel change, out of Resolver-session scope)

In pi-orchestrator.js, narrow the personaKeywords healthcare/architecture gate so bare compliance no longer trips it alone — e.g. require co-occurrence with a real healthcare term: /(hipaa|cqc|clinical|care|emar|safeguarding)/i AND compliance, or maintain a route=devops/fintech/backend whitelist that bypasses the healthcare-persona downgrade check.

Action taken this Resolver session

  1. Reproduced the daemon's 6h window query directly on mission-control.db — confirmed 17/18.
  2. Attempted mc.js unblock 105398 --actor john to unstick the fix ticket itself. session-task-lock-gate.sh correctly refused the write: this session is PID-locked to MC #106388 only, and #105398 is a different task — cross-task write blocked as designed (BLOCKED: cross-task operation detected). No override marker was created, so the block stands and #105398 remains blocked, unchanged.
  3. Not executed (explicitly out of scope): unblocking #105398 and dispatching the one-line kernel fix to CodeCraft/FlowForge (route already = devops). Recommended as a fresh, explicitly-scoped follow-up task/session.
  4. Not filed (lower priority): a ticket for the #103917-children wrapper generator dropping pre-approved route/ownership (route=qa override bug) — distinct from the persona-gate issue above.

Evidence

~/system/evidence/106388/root-cause-analysis.md

RAG Queue Recovery — MC #106978

RAG Queue Recovery — MC #106978

Status

S1–S4 are complete. The S5 offline package was rebuilt by MC #107082 for the reviewed d38f... client successor and exact 508e... server. It remains offline, has no authority receipt or UTC window, and is unauthorized for production effects. S6 requires a 19-day recurrence soak after a separately authorized successful G4; S7 closes the postmortem only after Proveo PASS.

Data-accounting result

Frozen S5 inputs

Independent round-13 QA, a separate reliability witness, native task-bound P2P, and clean post-commit verification all passed for d38f... (294/294 tests under both umask 0022 and 0077). These are offline readiness verdicts, not deployment authority. The actual qualified topology remains local Docker through the Cloudflare tunnel; the Azure VM is deallocated and the historical Azure body below its supersession notice remains documentation drift.

G4 sequence

  1. Record exact CEO authority and a maximum 45-minute UTC control window. Construct the production adapter and collect fresh bounded read-only/descriptive preflight proofs through allowlisted filesystem/command runners; caller assertions are not proof. Bind collected time, method ledger and proof digest to immutable config and authority.
  2. Deploy the exact server image first; prove authentication, one bounded document-ID lookup, no enumeration, and deterministic rollback. Server-only and pre-client failure uses only pinned server rollback.
  3. Coherently deploy all staged runtime/helper bytes: the five changed paths (agents/hivemind/hivemind.js, lib/rag-db-writer-lock.js, lib/rag-outbox.js, tools/lightrag-auth-helper.js, and tools/rag-drain-worker.js) and their unchanged staged receipt/spool/schema/config/HiveMind dependencies. Publish the private schema-v3 receipts and preactivated candidate with descriptor, file, and parent-directory fsync.
  4. Only after the future authority is consumed, quarantine legacy system/config/.lightrag-token-cache.json if it is a UID-owned, mode-0600, single-link regular file. The runtime cache policy is exclusively state/runtime-auth/lightrag-token-cache.json under a mode-0700 parent; absence is valid and any present file must be UID-owned, mode 0600, single-link, descriptor-bound. Rollback restores both cache prestates byte-for-byte.
  5. Reconcile the 112 accepted rows using a worker capability exposing only exact accepted-row selection/status GET and selected-row transitions. It has no POST/helper-login/nudge/permit/spool/metrics or queued/failed/history/global capability. A separately instantiated read-only verifier captures aggregate protected fingerprints before the worker and after it exits; protected reads never pass through the worker.
  6. Start the watchdog and require a hard local-audit acknowledgement plus a side-effect-free HiveMind local-gate-ack.
  7. After the future receipt/window exists, generate exactly three private selector files, one each for Mission Control, filesystem, and BookStack. Keep the exact selector record keys. Recompute its delivery key from private inputs as SHA-256(source NUL source_id NUL content_hash), then reserve/verify the exact DB row against all four values before generation/consumption. The unchanged record therefore transitively commits source. Each also binds task #107020, authority digest and the maximum 45-minute window; never put selector or delivery identity in argv or public evidence.
  8. Start the persistent drain at no more than 20 POSTs per rolling 60 seconds, then restore the three source adapters serially. Every G4 operator/worker POST—server/reconciliation/restoration cache-miss login, nudge, Slack, canary, adapter, replay, and drain—uses one centrally supplied durable permit immediately before transport and shares state across restart; the 21st is impossible. Existing-token GETs spend no POST permit. This coverage claim does not extend to unrelated non-G4 tools. Local audit and HiveMind are hard; Slack is retry-soft only under the literal policy. The superseded legacy outbox-ingest job remains disabled.
  9. Replay the seven authoritative MC outcomes plus a mandatory final structured delta sweep.

Terminal semantics

Rollback

Final sweep, Slack, final authority checks, semantic/privacy evidence validation/write and close remain in one rollback-protected authority boundary; child timeouts are capped by remaining time. Any wrong hash/image/router, unauthorized method, accepted-only POST, queued/history/global mutation, activation or auth-cache-policy failure, writer overlap, SQLite failure, missing audit/HiveMind acknowledgement, canary reservation/consumption failure or duplicate, privacy leak, unexplained replay delta, or window expiry triggers rollback. Track mutation stage durably. Server-only/pre-client failure never demands failed DB sidecars. After client mutation, stop all restored jobs, descriptor-capture and preserve the failed main plus exactly whichever optional WAL/SHM files exist (never fabricate one or omit an observed one), restore exact f50e..., restore all prior staged-path bytes, both cache prestates, receipts, and the pinned prior server image, then prove descriptor, fsync, offline/job, router, and health fences. Never use the untrusted historical generation.

Monitoring and closure

After a separately authorized successful S5, Proveo owns S6 verification for the required 19-day recurrence interval. The soak repeatedly checks immutable snapshots, queue progress, one-writer topology, durable permit capacity, daemon health, corruption signatures, auth-cache privacy, and downstream canaries. Main task #106978 remains open until S6 and S7 pass without force. BookStack publication remains blocked/drifted; this local file is canonical until an authenticated publication is separately authorized and verified.

LightRAG Backup — retarget na lr106747-* volumene + fail-loud guard (#106946)

MC #106946 — LightRAG backup retarget

Status: ŽIVO od 2026-08-08; prvi produkcijski run nedjelja 2026-08-09 02:00Z PROŠAO.

Šta je promijenjeno: lightrag-backup.sh retargetovan sa nepostojećih volumena na stvarnih 7 lr106747-* volumena + pre-flight fail-loud guard (odbija run ako bilo koji ciljni volumen fali) — builder devops-dev, evidence: /Users/makinja/system/evidence/106946/lightrag-backup-retarget-20260808.md

Nezavisna verifikacija #106946 — John, 2026-08-12 23:15

Nedjeljni run 2026-08-09 02:00-02:04Z (izvod iz ~/system/logs/lightrag-backup.log)

[2026-08-09T02:00:04Z] === LightRAG backup 20260809-040004 starting === [2026-08-09T02:00:04Z] Pre-flight guard: checking 7 target volumes exist before touching any container... [2026-08-09T02:00:05Z] Pre-flight guard OK — all 7 target volumes present with expected data [2026-08-09T02:00:05Z] Stopping containers (graceful)... [2026-08-09T02:00:09Z] dumping lr106747-data ... [2026-08-09T02:02:08Z] verify lr106747-data: tar OK, 87328 entries [2026-08-09T02:02:08Z] dumping lr106747-kg ... [2026-08-09T02:02:08Z] verify lr106747-kg: tar OK, 1 entries [2026-08-09T02:02:08Z] dumping lr106747-cache ... [2026-08-09T02:02:08Z] verify lr106747-cache: tar OK, 1 entries [2026-08-09T02:02:08Z] dumping lr106747-neo4j-data ... [2026-08-09T02:02:26Z] verify lr106747-neo4j-data: tar OK, 129 entries [2026-08-09T02:02:26Z] dumping lr106747-neo4j-import ... [2026-08-09T02:02:26Z] verify lr106747-neo4j-import: tar OK, 1 entries [2026-08-09T02:02:26Z] dumping lr106747-neo4j-logs ... [2026-08-09T02:02:26Z] verify lr106747-neo4j-logs: tar OK, 6 entries [2026-08-09T02:02:26Z] dumping lr106747-neo4j-plugins ... [2026-08-09T02:02:27Z] verify lr106747-neo4j-plugins: tar OK, 2 entries [2026-08-09T02:02:27Z] Starting containers... [2026-08-09T02:02:38Z] Total downtime: 149s [2026-08-09T02:02:42Z] === Backup complete: /Users/makinja/system/backups/lightrag/20260809-040004 (1.5G) — structural check failures: 0 === [2026-08-09T02:02:57Z] Post-backup LightRAG health: starting [2026-08-09T02:02:57Z] Retention (local): 4 snapshots kept [2026-08-09T02:02:57Z] Uploading snapshot to Azure Blob (lightrag-backup)... "date": "2026-08-09T02:02:57+00:00", "lastModified": "2026-08-09T02:02:58+00:00", "date": "2026-08-09T02:03:51+00:00", "lastModified": "2026-08-09T02:03:51+00:00", "date": "2026-08-09T02:03:52+00:00", "lastModified": "2026-08-09T02:03:52+00:00", "date": "2026-08-09T02:04:01+00:00", "lastModified": "2026-08-09T02:04:02+00:00", "date": "2026-08-09T02:04:02+00:00", "lastModified": "2026-08-09T02:04:03+00:00", "date": "2026-08-09T02:04:03+00:00", "lastModified": "2026-08-09T02:04:04+00:00", "date": "2026-08-09T02:04:04+00:00", "lastModified": "2026-08-09T02:04:05+00:00", [2026-08-09T02:04:05Z] Azure upload OK — snapshot 20260809-040004 offsite at blob path 20260809-040004/ [2026-08-09T02:04:05Z] Azure rotation — pruning snapshots older than 8 most recent... [2026-08-09T02:04:06Z] Azure: deleting old snapshot 20260614-040005/ [2026-08-09T02:04:06Z] === Done ===

Health check danas (izvod iz ~/system/logs/backup-health-check.log)

[2026-08-12 21:16:19] SKIP: Azure blob check disabled by policy flag (/Users/makinja/system/config/backup-policy-local-only.flag) [2026-08-12 21:16:19] --- Checking ListBlobs egress (last 6h) --- [2026-08-12 21:16:20] OK: ListBlobs egress 0GB in last 6h (limit: 5GB) [2026-08-12 21:16:20] --- Checking ANVIL Docker containers --- [2026-08-12 21:16:21] OK: Container matching 'lightrag' is running [2026-08-12 21:16:21] OK: Container matching 'neo4j' is running [2026-08-12 21:16:21] OK: Container matching 'postgres' is running [2026-08-12 21:16:21] === Health Check SUMMARY === [2026-08-12 21:16:21] All checks PASSED. [2026-08-12 21:16:21] === Health Check COMPLETE ===

FORGE colima sizing — kad CI pada na OOM u docker buildu (MC #107134/#107139)

FORGE colima sizing — kad CI pada na OOM u docker buildu

Status: AKTIVNO · Nastalo: 2026-08-14, MC #107134/#107139 · Vrijedi za: self-hosted azdo agent bilko-forge-1 na FORGE (makinja@100.94.54.37)

Simptom

Build na main pada u fazi Build (linux/amd64 → ACR):

npm error signal SIGKILL
ResourceExhausted: cannot allocate memory

Sve prethodne faze prolaze — Kotlin compile, integrationTest, Kotest su zeleni. Pada isključivo korak koji gradi bilko-web docker image.

Nije isto što i MC #104886 („Reached heap limit"). Tamo je JS heap dosegao svoj strop; ovdje host/VM nema RAM-a, pa dizanje --max-old-space-size pogoršava stvar.

Uzrok

colima VM na FORGE-u je imao 6 GiB / 4 CPU na mašini od 256 GB. Uz to se bilko-web gradi kao docker buildx --platform linux/amd64, dakle QEMU emulacija na ARM-u, što dodatno diže potrošnju.

Popravka

ssh makinja@100.94.54.37
export PATH=/opt/homebrew/bin:$PATH
# backup postojeceg configa prije izmjene
cp ~/.colima/default/colima.yaml ~/.colima/default/colima.yaml.bak-$(date +%Y%m%d)
colima stop && colima start --cpu 8 --memory 24

Provjera na oba mjesta — ne vjerovati jednom:

colima list                        # -> aarch64, 8 CPU, 24GiB
docker info | grep -i "total memory"   # -> Total Memory: 23.42GiB

Kako znati da je stvarno riješeno

Ne po tome što VM ima više RAM-a, nego po kontrolisanom prije/poslije na istoj grani:

Build sha colima Ishod
1065, 1066, 1067, 1070 4f9224af 6 GiB svi pali na fazi 2, ~30 min
1072 e6f3a526 24 GiB faza 2 prošla — nekeširano, "Compiled successfully in 4.2min", bez SIGKILL
1073 e6f3a526 24 GiB faza 2 „prošla" za 16 s — korak je bio CACHED, nije izvršen

Primarni dokaz je build 1072, ne 1073. Ovo je ispravka nakon nezavisne peer-verifikacije (2026-08-15): 1073 je bio prvi citiran, ali njegov OOM-osjetljiv korak (npm run build --workspace=@bilko/web) je bio registry-cache pogodak (#12 CACHED) — cijeli job 16 sekundi. Sam po sebi ne dokazuje ništa o memoriji. Keširan je bio zato što je 1072 (isti sha, u redu 8 minuta ranije, ništa za ponovnu upotrebu) taj isti korak već odradio nekeširano i uspješno pod 24 GiB.

Pouka koja se ponavlja: „faza prošla" nije isto što i „faza se izvršila". Provjeri je li korak stvarno radio ili je uzet iz keša prije nego ga navedeš kao dokaz.

Dokaz: ~/system/evidence/107134/colima-fix-proven-build1073-2026-08-14.md + peer-verifikacija ~/system/evidence/107139/peer-verify-transcript-2026-08-15.md

Kontrola confounda — RIJEŠENA EKSPERIMENTOM (2026-08-15)

1072 je na sha e6f3a526, a padovi su na 4f9224af — različit sha je bio stvaran confound.

Prvo sam ga pokušao odbraniti argumentom: git diff 4f9224af..e6f3a526 dira 113 fajlova, nijedan CI/build config, samo aplikativni kod, 8103 dodane linije — dakle posao je teži, ne lakši. Cross-vendor recenzent (gpt-5.6-sol) je to odbio, s pravom: to je rezonovanje, nije kontrola.

Kontrola je onda izvedena — isti sha, mijenja se samo memorija:

commit 4f9224af   (tačno onaj koji je 4× pao pod 6 GiB)
colima: 8 CPU / 24GiB   (docker info: Total Memory 23.42GiB)
docker buildx build --platform linux/amd64 -f apps/web/Dockerfile ...
#27 DONE 372.1s          exit=0          real 6m43.219s
grep -ci "sigkill|cannot allocate memory|ResourceExhausted|killed"  →  0

Izvedeno van CI-ja, direktno na FORGE-u u scratch worktreeu. Namjerno se nije puštalo kroz main pipeline: to bi deployalo stari aplikativni kod u živo okruženje, a PR/branch putanja uopšte ne gradi ovaj image pa ne može poslužiti kao test.

Dokaz: ~/system/evidence/107139/same-sha-control-build-2026-08-15.log + same-sha-control-preflight-2026-08-15.txt

Ni to nije bila prava kontrola — druga runda recenzije

Cross-vendor recenzent je i ovo odbio, opet s pravom:

„The same-SHA run is not a memory-only control because CPU doubled and the restart removed orphan containers."

Tačno. Promjena je bila 6 GiB / 4 CPU → 24 GiB / 8 CPU, a restart je uz to pomeo 3 orphan Testcontainers postgres kontejnera i lumiscare-redis. Tri varijable, ne jedna. Prvi „kontrolni" run je dokazao da novo okruženje radi — ne i šta je od toga bilo presudno.

Izolacija — treći run, i ovaj je konačan

Isti sha, CPU ostavljen na novoj vrijednosti (8), orphan kontejneri već pometeni, spuštena samo memorija na 6 GiB:

colima: aarch64, 8 CPU, 6GiB
#27 307.3  npm error signal SIGKILL
ERROR: ... "npm run build --workspace=@bilko/web" did not complete successfully: cannot allocate memory
ResourceExhausted
real 5m38.358s   exit=102

Dokaz: ~/system/evidence/107139/mem6gib-cpu8-isolation-build-2026-08-15.log

Konačna tabela — dvije tačke koje se razlikuju SAMO u memoriji

sha CPU RAM orphani ishod
4f9224af 4 6 GiB prisutni pao ×4 (CI 1065/1066/1067/1070), ~30 min
4f9224af 8 6 GiB pometeni pao — SIGKILL, cannot allocate memory, 5m38s
4f9224af 8 24 GiB pometeni prošao — 6m43s, bez SIGKILL

Srednji red isključuje obje konkurentske hipoteze odjednom: CPU je na novoj, višoj vrijednosti i ne spašava; orphani su počišćeni i ne spašavaju. Ostaje memorija.

Zaključak: memorija je bila vezujuće ograničenje. Dokazano izolacijom, ne argumentom.

FORGE je poslije eksperimenta vraćen na 8 CPU / 24 GiB i to je provjereno (colima list + docker info → Total Memory: 23.42GiB).

Pouka o metodi — tri kruga, tri različite greške

  1. Prvo sam kao dokaz naveo run čiji je korak bio keširan (#12 CACHED, 16 s) — „faza prošla" nije „faza se izvršila".
  2. Zatim sam confound branio argumentom („113 fajlova čini posao težim") umjesto kontrolom.
  3. Zatim sam kontrolu nazvao „samo memorija" iako su se promijenile tri stvari.

Moj vlastiti peer-verifikator je dao 12/12 PASS i propustio sve troje. Model drugog vendora je oborio dva kruga zaredom i oba puta bio u pravu. Kad se više varijabli promijeni zajedno, jedini izlaz je pustiti isti ulaz kroz postavke koje se razlikuju u jednoj stvari.

Dvije zamke koje su nas koštale

  1. PR build ne dokazuje ništa o OOM-u. Build 1071 (PR 338) je bio zelen poslije popravke i djelovao kao dokaz — ali PR putanja uopšte ne gradi web image. Prethodni pad istog PR-a (1069) bio je na integrationTest baseline check, ne na memoriji. Dokaz je isključivo main build.
  2. Jedan paralelni slot org-wide. Ako ručno pokreneš build dok drugi već radi na istoj grani, drugi čeka — build 1073 je izgubio 21 minutu čekajući 1072. Prije ručnog pokretanja provjeri šta već teče. azdo-pr-requeue.sh kancelira build koji je running.

Trajni pravac (nije urađeno)

Web image se gradi emulirano. Organizacija već ima plaćen Microsoft-hosted paralelni slot (isHosted:true, PurchasedCount:1, totalMinutes:1800) koji stoji prazan — svih 11 jobova je zakucano na bilko-selfhosted. Premještanje samo image-build joba na hosted agenta daje nativni amd64 (bez QEMU) i skida kontenciju, bez ijednog dolara.

Ograničenje: bilko-demo-pg je IP-ograničen na FORGE, pa jobovi koji diraju bazu (Flyway, E2E) ne mogu na hosted agenta bez izmjene firewalla.

Detalji i cijene: ~/system/evidence/107134/ci-capacity-options-2026-08-14.md

Metodološka napomena

Prije ove popravke napravljene su tri pogrešne atribucije uzastopno (Sentry source mape, dupli typecheck pod QEMU, pa „nešto u aplikaciji"). Sve tri izmjene su bile korisne same po sebi, ali nijedna nije bila uzrok. Uzrok je nađen tek kad je neko pogledao koliko RAM-a VM uopšte ima — a to je stajalo neprovjereno jer je ranije prijavljeno „nemam SSH pristup FORGE-u", što nije bilo tačno: makinja@ radi.

Slack → mail fallback (MC #107104)

Slack → mail fallback (MC #107104)

Od 2026-08-18 ~/system/tools/slack.js više ne gubi poruku kad Slack odbije da je primi. Fallback je dodan centralno, u cmdSend, cmdReply i cmdSendBlocks, pa pokriva svih ~140 pozivalaca odjednom. Nijedan demon nije mijenjan.

Zašto je uopšte trebalo

Workspace alai-talk.slack.com (6 korisnika) je na besplatnom planu i udara u limit poruka. Greška message_limit_exceeded pojavljuje se u 41 log fajlu pod ~/system/logs, samo u ops-watchdog-error.log 985 puta. Zadnja datirana pojava je 2026-08-13; ništa ne objašnjava zašto je stalo, pa može proraditi bez najave.

Šteta nije bila samo „izgubljen alert":

demon šta je gubio
tldr-weekly-synthesis na Slack padu radi process.exit(1) i baca cijeli sedmični memo
email-reactor-digest gubi dnevni digest u cijelosti
tldr-watch, bilko-demo-uat, mail-boot-signal, lightrag-monitor samo alert; podaci prežive

Mail fallback je jednom već postojao — ceo-daily-digest.js ga je uspješno okinuo 2026-08-13 — pa je tiho uklonjen neispraćenom izmjenom 2026-08-16. Zato sada živi u slack.js, jedno mjesto, umjesto da se ponovo raspe po pozivaocima.

Ugovor: tri razdvojena stanja

Ovo je bio ključni dio dizajna. tldr-weekly-synthesis odlučuje hoće li uništiti svoj izlaz na osnovu odgovora, pa „isporučeno" i „nigdje isporučeno" moraju biti razlučivi.

izlaz značenje exit
✓ Sent to #kanal Slack primio. Tekst nepromijenjen u odnosu na prije. 0
⚠ SLACK_FAILED_MAIL_FALLBACK_SENT #kanal … (Message-ID …) isporučeno mailom 0
⚠ SLACK_FAILED_MAIL_FALLBACK_DEDUPED #kanal … isti alert već mailovan u zadnjih 6 h 0
ERROR: DELIVERY_FAILED_BOTH #kanal: … (stderr) nigdje isporučeno 1

Exit 0 znači isporučeno BILO GDJE. Zato postojeći pozivaoci koji gledaju samo exit kod nisu slomljeni. Ko treba tačno stanje, čita tekst — ne kod.

Detalji koji se lako pokvare

Kako je dokazano (uživo, ne simulirano)

  1. Slack uspio — pravi send na #john-digest → ✓ Sent to #john-digest, exit 0, mail se ne dira.
  2. Slack pao → mail stigao — prisiljeno stvarno nepostojećim kanalom (resolveChannel() je zaista bacio „Channel not found"). Mail potvrđen dvaput: u Sent folderu (uid 104) i IMAP pretragom u samom inboxu alem@alai.no (uid 3215), s pročitanim tijelom.
  3. Dedup — isti alert ponovljen odmah → DEDUPED, exit 0, drugi mail nije poslan.
  4. Oba pala — bogus kanal + SLACK_MAIL_FALLBACK_TO na domen bez MX zapisa → DELIVERY_FAILED_BOTH, exit 1.

Nije zasebno paljena samo require()-fail grana (brisanje žive produkcijske datoteke je bilo van opsega); dijeli isti agregacijski put dokazan u testu 4.

Dokazi: ~/system/evidence/107104/ (8 fajlova). Commit 955b8491a5.

Odluka o Slack Pro

CEO, 2026-08-18: ne uzimati Pro za sada. Greške su dormantne od 13.08., a s fallbackom Slack ispad više ne uništava podatke nego alerte preusmjeri na mail. Ušteda ≈ 486–594 EUR/god (€6,75/korisnik/mj godišnje ili €8,25 mjesečno × 6 mjesta). Ponovo procijeniti ako Slack opet stane.

Poznato otvoreno, namjerno nedirano

TLDR zero-input false PASS — MC #900160

TLDR zero-input false PASS — MC #900160

Incident

On 2026-08-23 the TLDR briefing/actionizer chain produced no actionable output. The absence of input is the incident signal; it must not be reported as a healthy run.

Verified current behavior

  1. com.john.tldr-briefing finds zero eligible emails, writes a JSON sentinel with reason: no_unbriefed_emails, and exits 0.
  2. com.john.tldr-actionizer reads zero insights, emits no result/task, and exits 0.
  3. com.john.tldr-watch maps the zero-input sentinel to VERDICT=PASS.

This is fail-open observability: missing input is converted into green health and the operator receives no useful run result.

Required outcome

Implemented result

Live commits: 854aa1a5f (fail-closed zero input) and a853184cc (publication-day schedule).

The three stages now fail closed on zero input:

Positive control with five insights still completes with exit 0. Independent Gemini review returned PASS with zero unresolved P0/P1 findings.

Debugged source and schedule root cause

The mail database and direct IMAP search agree: no TLDR message exists after Friday 2026-08-21. Historical delivery data over 120 days contains messages only Monday-Friday, normally arriving after the old 09:00 briefing slot. The jobs nevertheless ran every day, so Sunday and Monday were guaranteed zero-input runs.

The active schedule is now explicit:

This processes the prior Monday-Friday publication day. RunAtLoad was removed from briefing. On a scheduled Tuesday-Saturday run, missing input remains ZERO_INPUT_ERROR/FAIL; on Sunday/Monday no run is scheduled. Active plists were reloaded and launchd shows weekdays 2-6 with zero immediate Sunday runs.

Evidence