# Infrastructure Overview

# Runbook: Local Infrastructure

**Platform:** Mac Studio M3 Ultra, 96GB RAM, macOS
**Services:** Docker containers, LaunchAgents, Cloudflare tunnels

---

## Docker Services

### Status Check
```bash
docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'
```

### Services
| Container | Image | Port | Health |
|-----------|-------|------|--------|
| mattermost | mattermost/mattermost-enterprise | 8065 | healthcheck |
| mattermost-db | postgres:13 | 5432 (internal) | — |
| planka | ghcr.io/plankanban/planka | 3100→1337 | healthcheck |
| planka-db | postgres:15-alpine | 5433 (internal) | healthcheck |
| documenso | documenso/documenso | 3003 | — |
| documenso-db | postgres | 5434 (internal) | healthcheck |
| bookstack | lscr.io/linuxserver/bookstack | 6875→80 | — |
| bookstack_db | lscr.io/linuxserver/mariadb | 3306 (internal) | — |

### Restart a container
```bash
docker restart <container_name>
# Example: docker restart mattermost
```

### Restart all
```bash
# Mattermost stack
cd ~/system/services/mattermost && docker compose down && docker compose up -d

# Planka stack
cd ~/system/services/planka && docker compose down && docker compose up -d

# Documenso
cd ~/system/services/documenso && docker compose down && docker compose up -d

# BookStack
cd ~/system/services/bookstack && docker compose down && docker compose up -d
```

### View logs
```bash
docker logs <container_name> --tail 50
docker logs <container_name> -f  # follow
```

### Disk cleanup (if disk >90%)
```bash
docker system prune -f            # Remove unused images, containers, networks
docker volume prune -f             # Remove unused volumes (CAREFUL: data loss)
```

---

## Cloudflare Tunnels

### Config
```bash
cat ~/.cloudflared/config.yml
```

### Routes — local Mac tunnel (3315a609-7934-45c5-ad0c-56d86d16374d)
Verified live 2026-07-28 by reading `~/.cloudflared/config.yml` directly (not from memory).

| Hostname | Target | Service |
|----------|--------|---------|
| sign.basicconsulting.no | localhost:3003 | Documenso — **only basicconsulting.no ingress rule exists in this tunnel's config**; `sign.alai.no` still resolves live (HTTP 302, verified 2026-07-28) but has no matching ingress line here — routed some other way (separate DNS/Access rule not yet located). Needs FlowForge follow-up before assuming `sign.alai.no` is safe to rely on for Documenso incident response. |
| mm.basicconsulting.no | http://192.168.68.61:8065 | Mattermost — **stale entry**: service was decommissioned 2026-05-18 (see mattermost.md) but this ingress line was never removed from config.yml. Dead weight, not currently routable to anything since the LAN host is retired; safe to delete. |
| boards.alai.no | not present as its own ingress line | Planka — service-registry.md claims `boards.alai.no → localhost:3100` via this tunnel, but no `boards.*` hostname appears in `~/.cloudflared/config.yml` as read 2026-07-28. Either Planka is routed via a different tunnel/mechanism, or the registry entry is stale. Flagged for FlowForge verification, not asserted as fact either way. |
| ollama.alai.no / ollama.basicconsulting.no | localhost:11434 / http://10.0.0.2:11435 | Both hostnames present — this one IS a clean dual-domain migration example (see inline comments in config.yml, 2026-06-30). |
| lumiscare-*.alai.no / lumiscare-*.basicconsulting.no | localhost 8090/4001/4002/4003/4567 | Both hostnames present for every lumiscare subdomain — another clean dual-domain pair. |
| lobby.basicconsulting.no, lobby-api.basicconsulting.no, api.basicconsulting.no, drop-api.basicconsulting.no, mc.basicconsulting.no, auth.basicconsulting.no, track.basicconsulting.no, ssh.basicconsulting.no, vnc.basicconsulting.no, kenan-hot.basicconsulting.no, bilko-demo.basicconsulting.no, bilko-demo-api.basicconsulting.no | various localhost ports | **basicconsulting.no-only** — no alai.no equivalent ingress line exists for any of these in this tunnel's config as of 2026-07-28. `bilko-demo.basicconsulting.no` is additionally known-dead (orphaned GCP Cloud Run origin, MC #10440). The rest are unverified live/dead here — this is an inventory of what the migration (MC #9392, paused) has NOT yet touched, not a claim that they're broken. |

### Azure VM tunnel — BookStack, Vaultwarden, Grafana (separate tunnel, not in the file above)
BookStack, Vaultwarden, Documenso-adjacent docs/vault/grafana hosts live on the Azure VM (4.223.110.181), not this Mac. Live-curl comparison run 2026-07-28 (`curl -o /dev/null -w '%{http_code}'`):

| Service | alai.no host | Status | basicconsulting.no host | Status |
|---------|-------------|--------|--------------------------|--------|
| BookStack | docs.alai.no | 200 | docs.basicconsulting.no | 200 |
| Vaultwarden | vault.alai.no | 200 | vault.basicconsulting.no | 200 |
| Grafana | grafana.alai.no | 302 | grafana.basicconsulting.no | 302 |
| Planka | boards.alai.no | 200 | boards.basicconsulting.no | 200 |
| Documenso | sign.alai.no | 302 | sign.basicconsulting.no | 302 |

**Reality check on MC #9392 ("Migracija basicconsulting.no → alai.no za sve tunnel hostnames + Azure VM servisi"):** all 5 services still answer identically on BOTH domains. The migration is **not** a completed cutover — both DNS names are live in parallel, matching MC #9392's actual status (`paused`, priority L, owner pi-orchestrator, as of 2026-07-28). Treat `*.alai.no` as the canonical/primary name to use going forward (per `~/system/CLAUDE.md` and service-registry.md), but do NOT assume `*.basicconsulting.no` is dead for these 5 services during an incident — it currently is not. See also the BookStack page "ALAI Domain Migration — basicconsulting.no → alai.no" (id 2666, tagged `staleness: needs-review`) and `~/system/docs/published/hosting-migration-log-basicconsulting-bilko-io-2026-07-28.md` for the corporate-site (non-tunnel) side of this same basicconsulting.no/alai.no duality — do not conflate the two: that log covers the Cloudflare-Pages-hosted apex sites, this section covers the Azure-VM Cloudflare-Tunnel-hosted internal tools.

### Status
```bash
cloudflared tunnel info 3315a609-7934-45c5-ad0c-56d86d16374d
```

### Restart tunnel
```bash
# Tunnel runs as LaunchAgent
launchctl unload ~/Library/LaunchAgents/com.cloudflare.tunnel.plist
launchctl load ~/Library/LaunchAgents/com.cloudflare.tunnel.plist
```

---

## LaunchAgents (Daemons)

### List all custom daemons
```bash
launchctl list | grep -E "com\.(john|edita|cloudflare)"
```

### Expected daemons
| Daemon | Interval | Location |
|--------|----------|----------|
| com.john.ops-agent | 5 min | ~/Library/LaunchAgents/ |
| com.edita.autowork | 30 min | ~/Library/LaunchAgents/ |
| com.john.mc-dashboard | always | ~/Library/LaunchAgents/ |
| com.john.mc-session-worker | on events | ~/Library/LaunchAgents/ |
| com.john.legal-docs-azure-sync | daily 02:00 | ~/Library/LaunchAgents/ |

### Legal docs Azure sync

`com.john.legal-docs-azure-sync` runs `/Users/makinja/system/daemons/legal-docs-azure-sync.sh` and backs up ALAI corporate legal documents to Azure Blob container `corporate-docs`.

Source directory order, verified for MC #10229 on 2026-07-29:
1. `${LEGAL_DOCS_SOURCE}` if explicitly set, otherwise `/Users/makinja/business/ALAI-Holding-AS/legal`.
2. Legacy fallback: `/Users/makinja/ALAI/legal`.

The canonical source is `/Users/makinja/business/ALAI-Holding-AS/legal` after the Phase D tenant restructure. Do not point the daemon only at the old `~/ALAI/legal` path. The optional ISBD upload source remains best-effort and is skipped when absent.

Quick verification:
```bash
LEGAL_DOCS_AZURE_SYNC_DRY_RUN=true \
LEGAL_DOCS_AZURE_SYNC_LOG_DIR=/tmp/legal-docs-sync-test \
bash ~/system/daemons/legal-docs-azure-sync.sh
bash -n ~/system/daemons/legal-docs-azure-sync.sh
plutil -lint ~/projects/alai-system/config/launchagents/com.john.legal-docs-azure-sync.plist
```

### Load/unload
```bash
launchctl load ~/Library/LaunchAgents/<plist-name>.plist
launchctl unload ~/Library/LaunchAgents/<plist-name>.plist
```

---

## Ollama (Local AI)

### Status
```bash
curl -s http://localhost:11434/api/tags | python3 -c "import sys,json; [print(m['name']) for m in json.load(sys.stdin)['models']]"
```

### Models
| Model | Size | Use |
|-------|------|-----|
| llama3.1:8b | 5GB | Fast classification (ops-agent) |
| qwen2.5-coder:32b | 19GB | Code generation, contextual responses |
| llama3.1:70b | 40GB | Research, writing |

### Restart Ollama
```bash
# Ollama runs as macOS app
killall ollama 2>/dev/null
open -a Ollama
```

---

## Mission Control Dashboard

### Status
```bash
curl -s http://localhost:3030 | head -1
```

### Restart
```bash
launchctl unload ~/Library/LaunchAgents/com.john.mc-dashboard.plist
launchctl load ~/Library/LaunchAgents/com.john.mc-dashboard.plist
```

---

## Full Health Check

```bash
# Human-readable
node ~/system/tools/health-check.js

# JSON (programmatic)
node ~/system/tools/health-check.js --json

# Quick (HTTP only)
node ~/system/tools/health-check.js --quick
```

---

## After System Reboot

All LaunchAgents with `RunAtLoad: true` start automatically. Verify:

```bash
# 1. Check Docker is running
docker ps

# 2. Check all daemons
launchctl list | grep -E "com\.(john|edita|cloudflare)"

# 3. Run health check
node ~/system/tools/health-check.js

# 4. If anything missing, load it
launchctl load ~/Library/LaunchAgents/<missing>.plist
```

---

## Incident Procedure — Azure VM tunnel service unreachable (BookStack/Vault/Grafana/Planka/Documenso)

Because the basicconsulting.no → alai.no tunnel-hostname migration (MC #9392) is **paused, not complete**, both domain variants currently resolve for these 5 services. Do not skip a step just because one domain "looks fine."

1. **Confirm which hostname actually failed.** Check both variants before escalating:
   ```bash
   for h in docs vault grafana boards sign; do
     curl -s -o /dev/null -w "$h.alai.no=%{http_code}  " -m 5 "https://$h.alai.no"
     curl -s -o /dev/null -w "$h.basicconsulting.no=%{http_code}\n" -m 5 "https://$h.basicconsulting.no"
   done
   ```
   If only ONE domain variant fails, this is a DNS/Cloudflare-zone issue, not the origin (Azure VM) — the origin serves both from the same backend.
2. **If BOTH variants fail:** origin is down. SSH to the Azure VM (`ssh -i ~/.ssh/azure_alai alai-admin@4.223.110.181`) and check the relevant container per `~/system/context/docs/runbooks/infrastructure.md` Docker Services section / service-specific runbook.
3. **Do not "fix" a basicconsulting.no failure by declaring it decommissioned** — unlike mm.basicconsulting.no (actually retired), these 5 hosts are still live and serving real traffic; treat a basicconsulting.no-only failure as a real incident, not expected drift.
4. **Report which domain(s) failed** in the incident record — don't collapse alai.no/basicconsulting.no into one line item, since MC #9392 tracks them as distinct DNS surfaces still requiring cutover.

---

## Changelog

### 2026-07-28 — basicconsulting.no → alai.no tunnel migration audit (MC #106446, source #9392)
Live-verified (curl, both this Mac's `~/.cloudflared/config.yml` and the Azure VM's 5 tunnel-fronted services) that the MC #9392 migration is incomplete: BookStack/Vault/Grafana/Planka/Documenso all still answer on both `*.alai.no` and `*.basicconsulting.no`. Updated the Cloudflare Tunnels section above with the verified route inventory (including a stale `mm.basicconsulting.no` ingress line that should be deleted, and an unresolved discrepancy where `sign.alai.no`/`boards.alai.no` have no matching ingress rule in this tunnel's config yet still resolve). Added the incident procedure above for handling single-domain-only failures during the migration window. No production changes made — this is a documentation-only pass per MC #106446 scope; the actual tunnel-config cutover remains MC #9392 (paused).

### 2026-07-10 — LIGHTRAG_URL stale host fix (MC #105191)
`com.john.pi-orchestrator.plist` had `LIGHTRAG_URL` hardcoded to the dead host
`https://lightrag.basicconsulting.no/query` (times out), breaking `lightrag.js` calls
in sessions spawned under pi-orchestrator. Fixed to the live host
`https://lightrag.alai.no/query` (verified reachable, HTTP 405 on GET / — expected
for a POST-only API) in both live copies:
- `~/Library/LaunchAgents/com.john.pi-orchestrator.plist`
- `~/system/config/launchagents/com.john.pi-orchestrator.plist`

Also fixed the matching hardcoded fallback default in
`~/system/kernel/pi-orchestrator.js:2101` (`process.env.LIGHTRAG_URL || '...'`),
which pointed to the same dead host.

**Action required:** the already-running daemon (verify current pid via
`ps aux | grep pi-orchestrator`) was started before this fix and still holds the
stale value in its process environment (confirmed via `ps eww -p <pid> | grep
LIGHTRAG_URL`). It will only pick up the new value on next restart
(`launchctl unload/load ~/Library/LaunchAgents/com.john.pi-orchestrator.plist`
or a reboot). Restart was NOT performed as part of this fix — CEO should confirm
before bouncing a live daemon.

---

**Created:** 2026-02-10
**Last Updated:** 2026-07-28