ALAI Hosting Operations
ALAI Hosting Operations Runbook
Owner: FlowForge (Kelsey Hightower) | Updated: 2026-07-28 | MC: #8491, #10218, #10221
1. Overview
This runbook covers operational procedures for ALAI's static site hosting on Cloudflare Pages. For architecture and migration plan, see the ALAI Static Hosting Blueprint (Infrastructure chapter).
In Scope:
- Cloudflare Pages deployments (9 static sites)
- DNS configuration (Cloudflare DNS)
- SSL certificate management (auto-renewal)
- Rollback procedures (< 60s target)
- SENTINEL uptime monitoring integration
Out of Scope:
- Azure VM application services (BookStack, Documenso, Planka, Vaultwarden) — see individual runbooks
- GCP Cloud Run (Bilko API, Intesa demo) — see Bilko runbooks
- Dynamic Next.js apps (app.getdrop.no) — see Drop runbook
Exception: vm-alai-support host-level storage (data disk, Docker data-root) is documented in Section 7, since no dedicated Azure VM runbook exists yet.
2. Rollback Procedure
When: Deploy caused production issue (5xx errors, broken UI, functionality regression)
Target: < 60 seconds from decision to live rollback
Step 1: Identify Last Known Good Deployment
# List recent deployments
npx wrangler pages deployment list --project-name=<project-name>
# Example output:
# ID: abc123def456
# Created: 2026-04-20 14:30:00
# Branch: main
# Status: active
Step 2: Execute Rollback
NOTE — wrangler 4.x breaking change:
wrangler pages deployment rollbackwas removed in wrangler 4.x. The subcommand no longer exists and the/rollbackCF API endpoint returns 405 for direct-upload deployments. Do NOT use it. Use the CF API re-deploy path below. (Reference: wrangler upstream release notes; verified in Proveo pilot on basicconsulting.no, MC #8494. See also ALAI Static Hosting Blueprint Section 3, "DR: Restore Site in < 60 Seconds".)
# CF API re-deploy (replaces deprecated wrangler rollback) — use ID from step 1
export CF_API_TOKEN="<your-cloudflare-api-token>" # scope: Cloudflare Pages: Edit
export CF_ACCOUNT_ID="<your-cloudflare-account-id>"
export CF_PROJECT_NAME="<project-name>"
curl -s -X POST \
"https://api.cloudflare.com/client/v4/accounts/${CF_ACCOUNT_ID}/pages/projects/${CF_PROJECT_NAME}/deployments/<deployment-id>/retry" \
-H "Authorization: Bearer ${CF_API_TOKEN}" \
-H "Content-Type: application/json" | python3 -c "import sys,json; r=json.load(sys.stdin); print('OK —', r['result']['id']) if r['success'] else print('ERROR:', r['errors'])"
# Example:
# curl -s -X POST ".../pages/projects/alai-no/deployments/abc123def456/retry" -H "Authorization: Bearer ${CF_API_TOKEN}" ...
Step 3: Verify
# Check HTTP status
curl -I https://<domain>
# Expected: HTTP/2 200
# If 5xx persists → escalate to L2 (Kelsey)
Step 4: Alert & Document
# Post to Slack
node ~/system/tools/slack.js send "#infra-alerts" \
"ROLLBACK executed: <project-name> to deployment <deployment-id> at $(date -u +%Y-%m-%dT%H:%M:%SZ)"
# Create incident report (if > 5 min downtime)
node ~/system/tools/mc.js add "Incident: <domain> rollback" \
--desc "Reason: [fill]. Rollback target: [deployment-id]. Downtime: [X min]" \
--priority H --owner kelsey
3. SSL Certificate Auto-Renewal
Cloudflare Pages manages SSL certificates automatically via Cloudflare's CA. Certificates renew 30 days before expiry.
No manual action required.
Troubleshooting: SSL Cert Warning
If SENTINEL alerts "SSL cert expiry < 30 days":
# Step 1: Verify domain DNS points to Cloudflare
dig <domain> +short
# Expected: CNAME to <project-name>.pages.dev or Cloudflare IP range
# Step 2: Check Cloudflare dashboard
open "https://dash.cloudflare.com/pages"
# Navigate to: Project > Settings > Custom domains
# Verify: "SSL/TLS certificate" shows "Active"
# Step 3: If cert not renewing, trigger manual renewal
# (Cloudflare Pages does not expose manual renewal API — contact support)
node ~/system/tools/slack.js send "#infra-alerts" \
"SSL cert not auto-renewing for <domain> — escalating to Cloudflare support"
4. Migration Workflow: New Site
Input: New static site needs hosting (markdown, React, Next.js static export, Astro)
Output: Site live on custom domain with SSL, SENTINEL monitoring enabled
Step 1: Validate Static Export
# For Next.js: verify static export enabled
grep 'output.*export' /path/to/site/next.config.js
# Expected: output: 'export'
# Build locally to verify
cd /path/to/site && npm run build
# Expected: Output directory exists (out/, dist/, .next/)
Step 2: Create Cloudflare Pages Project
# Option A: Dashboard (recommended for first-time)
open "https://dash.cloudflare.com/pages"
# Click: Create a project > Connect to Git > Select repo
# Option B: CLI
npx wrangler pages project create <project-name> --production-branch main
Step 3: Configure Build Settings
In Cloudflare dashboard: Project > Settings > Builds
| Framework | Build command | Output directory |
|---|---|---|
| Static HTML | (none) | / |
| Next.js (static export) | npm run build |
out |
| Astro | npm run build |
dist |
Save settings.
Step 4: Add GitHub Actions Workflow
Copy from template:
cp /Users/makinja/system/specs/templates/cf-pages-deploy.yml \
/path/to/site/.github/workflows/deploy.yml
Commit and push to trigger first deploy.
Step 5: Add Custom Domain
# In Cloudflare dashboard: Project > Custom domains > Add custom domain
# Enter: <domain>
# If domain DNS is already on Cloudflare: CNAME record auto-created
# If domain DNS is external: Manual CNAME to <project-name>.pages.dev required
Verify SSL activates (usually < 5 min).
Step 6: Enable SENTINEL Monitoring
Add domain to /Users/makinja/system/tools/sentinel-uptime.sh:
# Open file
nano /Users/makinja/system/tools/sentinel-uptime.sh
# Add line to SITES array:
"https://<domain>"
# Save and test
bash /Users/makinja/system/tools/sentinel-uptime.sh
Verify Slack alert NOT sent (indicates site UP).
Step 7: Document
Update site inventory:
# Add line to ~/system/docs/infrastructure-inventory.md
echo "| <domain> | Cloudflare Pages | <project-name> | [GitHub repo URL] | ACTIVE |" \
>> ~/system/docs/infrastructure-inventory.md
5. SENTINEL Uptime Integration
SENTINEL checks all ALAI sites every 5 minutes via cron.
Script: /Users/makinja/system/tools/sentinel-uptime.sh
Cron: */5 * * * * bash /Users/makinja/system/tools/sentinel-uptime.sh
Alert Channel: #infra-alerts (Slack)
Add New Site to SENTINEL
# Edit SITES array
nano /Users/makinja/system/tools/sentinel-uptime.sh
# Add:
"https://<domain>"
# Test manually
bash /Users/makinja/system/tools/sentinel-uptime.sh
# Expected: No output (site UP) or Slack alert (site DOWN)
Troubleshoot False Alerts
If SENTINEL reports DOWN but site is UP:
# Test from command line
curl -I --max-time 10 https://<domain>
# If returns 200: SENTINEL script has timeout issue (increase --max-time)
# If returns 5xx: Real issue — investigate Cloudflare Pages logs
# If returns 301/302: Update SENTINEL to accept redirects
6. Emergency DR: Serve from Azure VM
Scenario: Cloudflare Pages is down (e.g., Cloudflare incident) AND site is business-critical (e.g., alai.no during client demo).
Target: Site accessible within 120 seconds.
Step 1: Copy Build Output to VM
# From local machine:
cd /path/to/site
npm run build
scp -r ./out [email protected]:/var/www/<site-name>
Step 2: Serve via Caddy
# SSH to VM
ssh -i ~/.ssh/azure_alai [email protected]
# Start Caddy reverse proxy
sudo caddy reverse-proxy --from <domain> --to localhost:8080 &
# Start simple HTTP server
cd /var/www/<site-name> && python3 -m http.server 8080 &
Step 3: Update DNS (if needed)
# If Cloudflare DNS is also down, update DNS to point to Azure VM IP
# This requires registrar access — NOT recommended unless multi-hour Cloudflare outage
Step 4: Monitor & Rollback
# Verify site accessible
curl -I https://<domain>
# When Cloudflare recovers: DNS auto-reverts (CNAME to .pages.dev still exists)
# Kill Caddy process on VM
sudo killall caddy
7. Azure VM Data Disk Migration — vm-alai-support (MC #10218)
What: vm-alai-support's OS disk (30GB, Standard_B2als_v2) was filling up from Docker/containerd data (8 co-located services). CEO decision (Opcija 2, 2026-04-29) approved adding a dedicated managed data disk and moving the Docker data-root off the OS disk. Executed by FlowForge (Kelsey Hightower) 2026-04-30.
Services affected (all on vm-alai-support): bookstack, documenso, planka, baikal, vaultwarden, grafana, prometheus, caddy.
7.1 Disk Spec (tool-verified 2026-07-28 via az disk show)
| Field | Value |
|---|---|
| Disk name | disk-alai-support-data |
| Resource group | rg-alai-support (subscription 5b0b4d9b-e677-464e-abf0-5170cbce3b8e) |
| Region | swedencentral |
| Size | 128 GB |
| SKU | StandardSSD_LRS (Standard SSD, tier "Standard") |
| Mount | /mnt/data (/dev/sda1, ext4) |
| Attached to | vm-alai-support |
| Created | 2026-04-30T03:33:33Z |
Docker daemon data-root |
/mnt/data/docker (/etc/docker/daemon.json) |
⚠️ Discrepancy vs. task record: MC #10218's title and DoD evidence describe this as "Premium SSD P10 128GB / Premium_LRS." Live Azure state (checked 2026-07-28) shows the actual disk is StandardSSD_LRS (Standard SSD "E10" 128GB tier), not Premium. Cost/perf implications: Standard SSD is cheaper and lower-IOPS than Premium P10 — if a Premium disk was genuinely intended, this is an open gap; if Standard was chosen deliberately for cost, the task title/evidence text was simply inaccurate. Not re-litigated here — flagging so nobody cites "Premium P10" as fact downstream.
Mechanism note: the task title says "update docker-compose.yml bind paths," but /opt/alai/docker-compose.yml uses named Docker volumes (e.g. bookstack_config, caddy_data), not host bind paths. The actual migration mechanism was repointing the Docker daemon's data-root to /mnt/data/docker in /etc/docker/daemon.json — compose files were not edited for path changes.
7.2 Pre-requisite (verified before migration)
Azure Backup (RSV) first recovery point confirmed to exist before any migration step ran (MC #10210):
- Recovery point ID
7702054437221953196, 2026-04-29 22:09Z,FileSystemConsistent. - Vault:
rsv-alai-support, containervm-alai-support.
7.3 Migration Steps (as executed)
- Confirm RSV recovery point exists (#10210, above).
az disk create --name disk-alai-support-data -g rg-alai-support ...(128GB).- Attach disk to vm-alai-support (LUN0 per DoD evidence).
- SSH to VM, format new disk ext4, mount at
/mnt/data. - Stop Docker/containerd.
rsync/var/lib/docker→/mnt/data/dockerand/var/lib/containerd→/mnt/data/containerd.- Set
"data-root": "/mnt/data/docker"in/etc/docker/daemon.json. - Start Docker/containerd; verify all 8 services (20 containers) come up healthy.
- Log phase completion:
~/system/specs/system-reform-status.log(DATA-DISK-MIGRATION | STARTED | MC:#10218). mc.js ready 10218with DoD evidence; task marked done 2026-04-30 10:13:51.
Result (DoD evidence, MC #10218): 20 containers healthy post-migration. Old /var/lib/docker and /var/lib/containerd left in place pending a follow-up cleanup task (zero-FD check required first).
7.4 Old Directory Cleanup (7-day deletion window, MC #10228)
- Target:
/var/lib/containerd(16G) +/var/lib/docker(2.5G) = 18.5G to reclaim. OS disk was at 94% at time of ticket. - Gate: zero open file descriptors on the old dirs confirmed before deletion (safety check — nothing still had them open post-migration).
- Deadline: 2026-05-07 (MC #10228
Deadlinefield). (Task #10221's brief cites "~2026-05-06" as an estimate; the actual tracked deadline in MC is 2026-05-07.) - Completed: 2026-05-05 15:36:11 — 2 days ahead of deadline.
Live verification (2026-07-28): /var/lib/docker and /var/lib/containerd no longer exist on vm-alai-support — cleanup confirmed still in effect.
7.5 Current State (tool-verified 2026-07-28)
$ df -h / /mnt/data
Filesystem Size Used Avail Use% Mounted on
/dev/root 29G 11G 19G 36% /
/dev/sda1 126G 19G 101G 16% /mnt/data
$ docker info | grep "Docker Root Dir"
Docker Root Dir: /mnt/data/docker
$ docker ps | wc -l
20
OS disk usage dropped from a pre-migration ~94% down to 36% (11G/29G) after the data-root move + old-dir cleanup. 20 containers running, matching the post-migration DoD count.
7.6 Rollback Procedure
No pre-written rollback script exists for this migration; the safety net during the 7-day window (before #10228 cleanup) was the untouched copy of /var/lib/docker and /var/lib/containerd on the OS disk plus the confirmed Azure Backup RSV recovery point (Section 7.2). After cleanup (2026-05-05), the OS-disk copies no longer exist — rollback today means either restoring from the Azure Backup RSV recovery point, or reversing the data-root pointer if /mnt/data itself is the failure:
A. Data disk corrupted / unmountable (VM boots, OS disk fine):
ssh -i ~/.ssh/azure_alai [email protected]
sudo systemctl stop docker
# point data-root back to local OS disk (temporary, degraded — no persisted volumes)
sudo sed -i 's#/mnt/data/docker#/var/lib/docker#' /etc/docker/daemon.json
sudo systemctl start docker
# Services will start with EMPTY volumes — this is a stop-gap to get containers running,
# not a data recovery step. Follow with Option B to recover actual data.
B. Full VM/data restore (disk lost, or data corruption needing point-in-time recovery):
# Restore from Azure Backup RSV (rsv-alai-support) to the recovery point closest
# to the desired time, or a later automatic RP if scheduled backups exist post-2026-04-29.
az backup restore restore-disks \
--resource-group rg-alai-support --vault-name rsv-alai-support \
--container-name "IaasVMContainer;iaasvmcontainerv2;rg-alai-support;vm-alai-support" \
--item-name "VM;iaasvmcontainerv2;rg-alai-support;vm-alai-support" \
--rp-name <recovery-point-name> --storage-account <staging-storage-account>
# Then attach recovered data disk to vm-alai-support, verify /mnt/data mount and
# docker data-root, restart services, verify all 8 services healthy (curl each domain).
Verify after any rollback:
curl -sI https://docs.alai.no | head -1 # BookStack
curl -sI https://vault.alai.no | head -1 # Vaultwarden
curl -sI https://sign.alai.no | head -1 # Documenso
curl -sI https://grafana.alai.no | head -1
curl -sI https://boards.alai.no | head -1 # Planka
8. Escalation
| Issue | L1 Action | L2 Escalation | L3 Escalation |
|---|---|---|---|
| Deploy failure | Review build logs; check package.json/next.config.js | Kelsey investigates Cloudflare Pages logs | Contact Cloudflare support via dashboard |
| 5xx errors (< 5 min) | Execute rollback (Section 2) | Kelsey reviews last commit for breaking change | CEO notification + DR activation (Section 6) |
| SSL cert not renewing | Verify DNS (Section 3) | Kelsey triggers manual renewal or contacts CF support | Switch to Let's Encrypt via Azure VM |
| SENTINEL false alerts | Verify site UP via curl; adjust timeout | Kelsey reviews SENTINEL script logic | Disable SENTINEL for that site; use external monitor |
| DNS not resolving | Verify Cloudflare DNS records; check registrar NS | Kelsey checks registrar portal for NS change | Contact registrar support |
Key Contacts:
- L2: Kelsey Hightower (FlowForge agent) via MC task
- L3: CEO (Alem Basic) via Slack DM or phone (+47 404 74 251)
9. Maintenance Schedule
| Task | Frequency | Owner | How |
|---|---|---|---|
| Test rollback procedure | Monthly | Proveo (Angie Jones) | Execute rollback on staging site; verify < 60s |
| Review SENTINEL alerts | Weekly | Kelsey | Check Slack #infra-alerts for false positives |
| Update dependency versions | Weekly | Renovate bot | Auto-merge minor/patch; manual review major |
| Backup DNS zone config | Weekly | Automated cron | Exports to ~/system/backups/dns/ |
| Verify SSL certs valid | Daily | SENTINEL | Auto-alert if < 30 days to expiry |
10. Related Docs
- ALAI Static Hosting Blueprint — Architecture & migration plan
- Infrastructure Inventory — All ALAI sites & services
- SENTINEL Reliability Sprint — Monitoring architecture
- Incident Response Playbook — General incident workflow
11. Change Log
| Date | Change | Author |
|---|---|---|
| 2026-04-20 | Initial version — rollback, SSL, migration, SENTINEL | Skillforge (MC #8491) |
| 2026-07-28 | Added Section 7 (Azure VM Data Disk Migration — vm-alai-support, MC #10218); flagged Premium_LRS vs. live StandardSSD_LRS discrepancy; renumbered Sections 8–11 | Skillforge (MC #10221) |
| 2026-07-28 | Section 2 Step 2: replaced deprecated wrangler pages deployment rollback (removed in wrangler 4.x) with CF API re-deploy path, matching the blueprint fix already applied under MC #8494 |
FlowForge (MC #8494) |
No comments to display
No comments to display