Infrastructure Overview

Runbook: Local Infrastructure

Platform: Mac Studio M3 Ultra, 96GB RAM, macOS Services: Docker containers, LaunchAgents, Cloudflare tunnels

Docker Services

Status Check

docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'

Services

Container Image Port Health
mattermost mattermost/mattermost-enterprise 8065 healthcheck
mattermost-db postgres:13 5432 (internal) —
planka ghcr.io/plankanban/planka 3100→1337 healthcheck
planka-db postgres:15-alpine 5433 (internal) healthcheck
documenso documenso/documenso 3003 —
documenso-db postgres 5434 (internal) healthcheck
bookstack lscr.io/linuxserver/bookstack 6875→80 —
bookstack_db lscr.io/linuxserver/mariadb 3306 (internal) —

Restart a container

docker restart <container_name>
# Example: docker restart mattermost

Restart all

# Mattermost stack
cd ~/system/services/mattermost && docker compose down && docker compose up -d

# Planka stack
cd ~/system/services/planka && docker compose down && docker compose up -d

# Documenso
cd ~/system/services/documenso && docker compose down && docker compose up -d

# BookStack
cd ~/system/services/bookstack && docker compose down && docker compose up -d

View logs

docker logs <container_name> --tail 50
docker logs <container_name> -f  # follow

Disk cleanup (if disk >90%)

docker system prune -f            # Remove unused images, containers, networks
docker volume prune -f             # Remove unused volumes (CAREFUL: data loss)

Cloudflare Tunnels

Config

cat ~/.cloudflared/config.yml

Routes — local Mac tunnel (3315a609-7934-45c5-ad0c-56d86d16374d)

Verified live 2026-07-28 by reading ~/.cloudflared/config.yml directly (not from memory).

Hostname Target Service
sign.basicconsulting.no localhost:3003 Documenso — only basicconsulting.no ingress rule exists in this tunnel's config; sign.alai.no still resolves live (HTTP 302, verified 2026-07-28) but has no matching ingress line here — routed some other way (separate DNS/Access rule not yet located). Needs FlowForge follow-up before assuming sign.alai.no is safe to rely on for Documenso incident response.
mm.basicconsulting.no http://192.168.68.61:8065 Mattermost — stale entry: service was decommissioned 2026-05-18 (see mattermost.md) but this ingress line was never removed from config.yml. Dead weight, not currently routable to anything since the LAN host is retired; safe to delete.
boards.alai.no not present as its own ingress line Planka — service-registry.md claims boards.alai.no → localhost:3100 via this tunnel, but no boards.* hostname appears in ~/.cloudflared/config.yml as read 2026-07-28. Either Planka is routed via a different tunnel/mechanism, or the registry entry is stale. Flagged for FlowForge verification, not asserted as fact either way.
ollama.alai.no / ollama.basicconsulting.no localhost:11434 / http://10.0.0.2:11435 Both hostnames present — this one IS a clean dual-domain migration example (see inline comments in config.yml, 2026-06-30).
lumiscare-.alai.no / lumiscare-.basicconsulting.no localhost 8090/4001/4002/4003/4567 Both hostnames present for every lumiscare subdomain — another clean dual-domain pair.
lobby.basicconsulting.no, lobby-api.basicconsulting.no, api.basicconsulting.no, drop-api.basicconsulting.no, mc.basicconsulting.no, auth.basicconsulting.no, track.basicconsulting.no, ssh.basicconsulting.no, vnc.basicconsulting.no, kenan-hot.basicconsulting.no, bilko-demo.basicconsulting.no, bilko-demo-api.basicconsulting.no various localhost ports basicconsulting.no-only — no alai.no equivalent ingress line exists for any of these in this tunnel's config as of 2026-07-28. bilko-demo.basicconsulting.no is additionally known-dead (orphaned GCP Cloud Run origin, MC #10440). The rest are unverified live/dead here — this is an inventory of what the migration (MC #9392, paused) has NOT yet touched, not a claim that they're broken.

Azure VM tunnel — BookStack, Vaultwarden, Grafana (separate tunnel, not in the file above)

BookStack, Vaultwarden, Documenso-adjacent docs/vault/grafana hosts live on the Azure VM (4.223.110.181), not this Mac. Live-curl comparison run 2026-07-28 (curl -o /dev/null -w '%{http_code}'):

Service alai.no host Status basicconsulting.no host Status
BookStack docs.alai.no 200 docs.basicconsulting.no 200
Vaultwarden vault.alai.no 200 vault.basicconsulting.no 200
Grafana grafana.alai.no 302 grafana.basicconsulting.no 302
Planka boards.alai.no 200 boards.basicconsulting.no 200
Documenso sign.alai.no 302 sign.basicconsulting.no 302

Reality check on MC #9392 ("Migracija basicconsulting.no → alai.no za sve tunnel hostnames + Azure VM servisi"): all 5 services still answer identically on BOTH domains. The migration is not a completed cutover — both DNS names are live in parallel, matching MC #9392's actual status (paused, priority L, owner pi-orchestrator, as of 2026-07-28). Treat *.alai.no as the canonical/primary name to use going forward (per ~/system/CLAUDE.md and service-registry.md), but do NOT assume *.basicconsulting.no is dead for these 5 services during an incident — it currently is not. See also the BookStack page "ALAI Domain Migration — basicconsulting.no → alai.no" (id 2666, tagged staleness: needs-review) and ~/system/docs/published/hosting-migration-log-basicconsulting-bilko-io-2026-07-28.md for the corporate-site (non-tunnel) side of this same basicconsulting.no/alai.no duality — do not conflate the two: that log covers the Cloudflare-Pages-hosted apex sites, this section covers the Azure-VM Cloudflare-Tunnel-hosted internal tools.

Status

cloudflared tunnel info 3315a609-7934-45c5-ad0c-56d86d16374d

Restart tunnel

# Tunnel runs as LaunchAgent
launchctl unload ~/Library/LaunchAgents/com.cloudflare.tunnel.plist
launchctl load ~/Library/LaunchAgents/com.cloudflare.tunnel.plist

LaunchAgents (Daemons)

List all custom daemons

launchctl list | grep -E "com\.(john|edita|cloudflare)"

Expected daemons

Daemon Interval Location
com.john.ops-agent 5 min ~/Library/LaunchAgents/
com.edita.autowork 30 min ~/Library/LaunchAgents/
com.john.mc-dashboard always ~/Library/LaunchAgents/
com.john.mc-session-worker on events ~/Library/LaunchAgents/
com.john.legal-docs-azure-sync daily 02:00 ~/Library/LaunchAgents/

com.john.legal-docs-azure-sync runs /Users/makinja/system/daemons/legal-docs-azure-sync.sh and backs up ALAI corporate legal documents to Azure Blob container corporate-docs.

Source directory order, verified for MC #10229 on 2026-07-29:

  1. ${LEGAL_DOCS_SOURCE} if explicitly set, otherwise /Users/makinja/business/ALAI-Holding-AS/legal.
  2. Legacy fallback: /Users/makinja/ALAI/legal.

The canonical source is /Users/makinja/business/ALAI-Holding-AS/legal after the Phase D tenant restructure. Do not point the daemon only at the old ~/ALAI/legal path. The optional ISBD upload source remains best-effort and is skipped when absent.

Quick verification:

Load/unload

launchctl load ~/Library/LaunchAgents/<plist-name>.plist
launchctl unload ~/Library/LaunchAgents/<plist-name>.plist

Ollama (Local AI)

Status

curl -s http://localhost:11434/api/tags | python3 -c "import sys,json; [print(m['name']) for m in json.load(sys.stdin)['models']]"

Models

Model Size Use
llama3.1:8b 5GB Fast classification (ops-agent)
qwen2.5-coder:32b 19GB Code generation, contextual responses
llama3.1:70b 40GB Research, writing

Restart Ollama

# Ollama runs as macOS app
killall ollama 2>/dev/null
open -a Ollama

Mission Control Dashboard

Status

curl -s http://localhost:3030 | head -1

Restart

launchctl unload ~/Library/LaunchAgents/com.john.mc-dashboard.plist
launchctl load ~/Library/LaunchAgents/com.john.mc-dashboard.plist

Full Health Check

# Human-readable
node ~/system/tools/health-check.js

# JSON (programmatic)
node ~/system/tools/health-check.js --json

# Quick (HTTP only)
node ~/system/tools/health-check.js --quick

After System Reboot

All LaunchAgents with RunAtLoad: true start automatically. Verify:

# 1. Check Docker is running
docker ps

# 2. Check all daemons
launchctl list | grep -E "com\.(john|edita|cloudflare)"

# 3. Run health check
node ~/system/tools/health-check.js

# 4. If anything missing, load it
launchctl load ~/Library/LaunchAgents/<missing>.plist

Incident Procedure — Azure VM tunnel service unreachable (BookStack/Vault/Grafana/Planka/Documenso)

Because the basicconsulting.no → alai.no tunnel-hostname migration (MC #9392) is paused, not complete, both domain variants currently resolve for these 5 services. Do not skip a step just because one domain "looks fine."

  1. Confirm which hostname actually failed. Check both variants before escalating: 
    for h in docs vault grafana boards sign; do
  curl -s -o /dev/null -w "$h.alai.no=%{http_code}  " -m 5 "https://$h.alai.no"
  curl -s -o /dev/null -w "$h.basicconsulting.no=%{http_code}\n" -m 5 "https://$h.basicconsulting.no"
done
    If only ONE domain variant fails, this is a DNS/Cloudflare-zone issue, not the origin (Azure VM) — the origin serves both from the same backend.
  2. If BOTH variants fail: origin is down. SSH to the Azure VM (ssh -i ~/.ssh/azure_alai alai-admin@4.223.110.181) and check the relevant container per ~/system/context/docs/runbooks/infrastructure.md Docker Services section / service-specific runbook.
  3. Do not "fix" a basicconsulting.no failure by declaring it decommissioned — unlike mm.basicconsulting.no (actually retired), these 5 hosts are still live and serving real traffic; treat a basicconsulting.no-only failure as a real incident, not expected drift.
  4. Report which domain(s) failed in the incident record — don't collapse alai.no/basicconsulting.no into one line item, since MC #9392 tracks them as distinct DNS surfaces still requiring cutover.

Changelog

2026-07-28 — basicconsulting.no → alai.no tunnel migration audit (MC #106446, source #9392)

Live-verified (curl, both this Mac's ~/.cloudflared/config.yml and the Azure VM's 5 tunnel-fronted services) that the MC #9392 migration is incomplete: BookStack/Vault/Grafana/Planka/Documenso all still answer on both *.alai.no and *.basicconsulting.no. Updated the Cloudflare Tunnels section above with the verified route inventory (including a stale mm.basicconsulting.no ingress line that should be deleted, and an unresolved discrepancy where sign.alai.no/boards.alai.no have no matching ingress rule in this tunnel's config yet still resolve). Added the incident procedure above for handling single-domain-only failures during the migration window. No production changes made — this is a documentation-only pass per MC #106446 scope; the actual tunnel-config cutover remains MC #9392 (paused).

2026-07-10 — LIGHTRAG_URL stale host fix (MC #105191)

com.john.pi-orchestrator.plist had LIGHTRAG_URL hardcoded to the dead host https://lightrag.basicconsulting.no/query (times out), breaking lightrag.js calls in sessions spawned under pi-orchestrator. Fixed to the live host https://lightrag.alai.no/query (verified reachable, HTTP 405 on GET / — expected for a POST-only API) in both live copies:

Also fixed the matching hardcoded fallback default in ~/system/kernel/pi-orchestrator.js:2101 (process.env.LIGHTRAG_URL || '...'), which pointed to the same dead host.

Action required: the already-running daemon (verify current pid via ps aux | grep pi-orchestrator) was started before this fix and still holds the stale value in its process environment (confirmed via ps eww -p <pid> | grep LIGHTRAG_URL). It will only pick up the new value on next restart (launchctl unload/load ~/Library/LaunchAgents/com.john.pi-orchestrator.plist or a reboot). Restart was NOT performed as part of this fix — CEO should confirm before bouncing a live daemon.

Created: 2026-02-10 Last Updated: 2026-07-28

Revision #10
Created 2026-02-20 09:38:28 UTC by John
Updated 2026-07-29 01:10:51 UTC by John