# ALAI Hosting Operations

# ALAI Hosting Operations Runbook

**Owner:** FlowForge (Kelsey Hightower) | **Updated:** 2026-07-28 | **MC:** #8491, #10218, #10221

---

## 1. Overview

This runbook covers operational procedures for ALAI's static site hosting on Cloudflare Pages. For architecture and migration plan, see the [ALAI Static Hosting Blueprint](https://docs.alai.no) (Infrastructure chapter).

**In Scope:**
- Cloudflare Pages deployments (9 static sites)
- DNS configuration (Cloudflare DNS)
- SSL certificate management (auto-renewal)
- Rollback procedures (< 60s target)
- SENTINEL uptime monitoring integration

**Out of Scope:**
- Azure VM *application* services (BookStack, Documenso, Planka, Vaultwarden) — see individual runbooks
- GCP Cloud Run (Bilko API, Intesa demo) — see Bilko runbooks
- Dynamic Next.js apps (app.getdrop.no) — see Drop runbook

**Exception:** vm-alai-support host-level storage (data disk, Docker data-root) is documented in Section 7, since no dedicated Azure VM runbook exists yet.

---

## 2. Rollback Procedure

**When:** Deploy caused production issue (5xx errors, broken UI, functionality regression)

**Target:** < 60 seconds from decision to live rollback

### Step 1: Identify Last Known Good Deployment

```bash
# List recent deployments
npx wrangler pages deployment list --project-name=<project-name>

# Example output:
# ID: abc123def456
# Created: 2026-04-20 14:30:00
# Branch: main
# Status: active
```

### Step 2: Execute Rollback

> **NOTE — wrangler 4.x breaking change:** `wrangler pages deployment rollback` was removed in wrangler 4.x. The subcommand no longer exists and the `/rollback` CF API endpoint returns 405 for direct-upload deployments. Do NOT use it. Use the CF API re-deploy path below. (Reference: wrangler upstream release notes; verified in Proveo pilot on basicconsulting.no, MC #8494. See also [ALAI Static Hosting Blueprint](https://docs.alai.no) Section 3, "DR: Restore Site in < 60 Seconds".)

```bash
# CF API re-deploy (replaces deprecated wrangler rollback) — use ID from step 1
export CF_API_TOKEN="<your-cloudflare-api-token>"   # scope: Cloudflare Pages: Edit
export CF_ACCOUNT_ID="<your-cloudflare-account-id>"
export CF_PROJECT_NAME="<project-name>"

curl -s -X POST \
  "https://api.cloudflare.com/client/v4/accounts/${CF_ACCOUNT_ID}/pages/projects/${CF_PROJECT_NAME}/deployments/<deployment-id>/retry" \
  -H "Authorization: Bearer ${CF_API_TOKEN}" \
  -H "Content-Type: application/json" | python3 -c "import sys,json; r=json.load(sys.stdin); print('OK —', r['result']['id']) if r['success'] else print('ERROR:', r['errors'])"

# Example:
# curl -s -X POST ".../pages/projects/alai-no/deployments/abc123def456/retry" -H "Authorization: Bearer ${CF_API_TOKEN}" ...
```

### Step 3: Verify

```bash
# Check HTTP status
curl -I https://<domain>

# Expected: HTTP/2 200
# If 5xx persists → escalate to L2 (Kelsey)
```

### Step 4: Alert & Document

```bash
# Post to Slack
node ~/system/tools/slack.js send "#infra-alerts" \
  "ROLLBACK executed: <project-name> to deployment <deployment-id> at $(date -u +%Y-%m-%dT%H:%M:%SZ)"

# Create incident report (if > 5 min downtime)
node ~/system/tools/mc.js add "Incident: <domain> rollback" \
  --desc "Reason: [fill]. Rollback target: [deployment-id]. Downtime: [X min]" \
  --priority H --owner kelsey
```

---

## 3. SSL Certificate Auto-Renewal

Cloudflare Pages manages SSL certificates automatically via Cloudflare's CA. Certificates renew 30 days before expiry.

**No manual action required.**

### Troubleshooting: SSL Cert Warning

If SENTINEL alerts "SSL cert expiry < 30 days":

```bash
# Step 1: Verify domain DNS points to Cloudflare
dig <domain> +short

# Expected: CNAME to <project-name>.pages.dev or Cloudflare IP range

# Step 2: Check Cloudflare dashboard
open "https://dash.cloudflare.com/pages"
# Navigate to: Project > Settings > Custom domains
# Verify: "SSL/TLS certificate" shows "Active"

# Step 3: If cert not renewing, trigger manual renewal
# (Cloudflare Pages does not expose manual renewal API — contact support)
node ~/system/tools/slack.js send "#infra-alerts" \
  "SSL cert not auto-renewing for <domain> — escalating to Cloudflare support"
```

---

## 4. Migration Workflow: New Site

**Input:** New static site needs hosting (markdown, React, Next.js static export, Astro)

**Output:** Site live on custom domain with SSL, SENTINEL monitoring enabled

### Step 1: Validate Static Export

```bash
# For Next.js: verify static export enabled
grep 'output.*export' /path/to/site/next.config.js

# Expected: output: 'export'

# Build locally to verify
cd /path/to/site && npm run build

# Expected: Output directory exists (out/, dist/, .next/)
```

### Step 2: Create Cloudflare Pages Project

```bash
# Option A: Dashboard (recommended for first-time)
open "https://dash.cloudflare.com/pages"
# Click: Create a project > Connect to Git > Select repo

# Option B: CLI
npx wrangler pages project create <project-name> --production-branch main
```

### Step 3: Configure Build Settings

In Cloudflare dashboard: Project > Settings > Builds

| Framework | Build command | Output directory |
|-----------|--------------|------------------|
| Static HTML | (none) | / |
| Next.js (static export) | `npm run build` | `out` |
| Astro | `npm run build` | `dist` |

Save settings.

### Step 4: Add GitHub Actions Workflow

Copy from template:

```bash
cp /Users/makinja/system/specs/templates/cf-pages-deploy.yml \
   /path/to/site/.github/workflows/deploy.yml
```

Commit and push to trigger first deploy.

### Step 5: Add Custom Domain

```bash
# In Cloudflare dashboard: Project > Custom domains > Add custom domain
# Enter: <domain>

# If domain DNS is already on Cloudflare: CNAME record auto-created
# If domain DNS is external: Manual CNAME to <project-name>.pages.dev required
```

Verify SSL activates (usually < 5 min).

### Step 6: Enable SENTINEL Monitoring

Add domain to `/Users/makinja/system/tools/sentinel-uptime.sh`:

```bash
# Open file
nano /Users/makinja/system/tools/sentinel-uptime.sh

# Add line to SITES array:
"https://<domain>"

# Save and test
bash /Users/makinja/system/tools/sentinel-uptime.sh
```

Verify Slack alert NOT sent (indicates site UP).

### Step 7: Document

Update site inventory:

```bash
# Add line to ~/system/docs/infrastructure-inventory.md
echo "| <domain> | Cloudflare Pages | <project-name> | [GitHub repo URL] | ACTIVE |" \
  >> ~/system/docs/infrastructure-inventory.md
```

---

## 5. SENTINEL Uptime Integration

SENTINEL checks all ALAI sites every 5 minutes via cron.

**Script:** `/Users/makinja/system/tools/sentinel-uptime.sh`

**Cron:** `*/5 * * * * bash /Users/makinja/system/tools/sentinel-uptime.sh`

**Alert Channel:** `#infra-alerts` (Slack)

### Add New Site to SENTINEL

```bash
# Edit SITES array
nano /Users/makinja/system/tools/sentinel-uptime.sh

# Add:
"https://<domain>"

# Test manually
bash /Users/makinja/system/tools/sentinel-uptime.sh

# Expected: No output (site UP) or Slack alert (site DOWN)
```

### Troubleshoot False Alerts

If SENTINEL reports DOWN but site is UP:

```bash
# Test from command line
curl -I --max-time 10 https://<domain>

# If returns 200: SENTINEL script has timeout issue (increase --max-time)
# If returns 5xx: Real issue — investigate Cloudflare Pages logs
# If returns 301/302: Update SENTINEL to accept redirects
```

---

## 6. Emergency DR: Serve from Azure VM

**Scenario:** Cloudflare Pages is down (e.g., Cloudflare incident) AND site is business-critical (e.g., alai.no during client demo).

**Target:** Site accessible within 120 seconds.

### Step 1: Copy Build Output to VM

```bash
# From local machine:
cd /path/to/site
npm run build
scp -r ./out alai-admin@4.223.110.181:/var/www/<site-name>
```

### Step 2: Serve via Caddy

```bash
# SSH to VM
ssh -i ~/.ssh/azure_alai alai-admin@4.223.110.181

# Start Caddy reverse proxy
sudo caddy reverse-proxy --from <domain> --to localhost:8080 &

# Start simple HTTP server
cd /var/www/<site-name> && python3 -m http.server 8080 &
```

### Step 3: Update DNS (if needed)

```bash
# If Cloudflare DNS is also down, update DNS to point to Azure VM IP
# This requires registrar access — NOT recommended unless multi-hour Cloudflare outage
```

### Step 4: Monitor & Rollback

```bash
# Verify site accessible
curl -I https://<domain>

# When Cloudflare recovers: DNS auto-reverts (CNAME to .pages.dev still exists)
# Kill Caddy process on VM
sudo killall caddy
```

---

## 7. Azure VM Data Disk Migration — vm-alai-support (MC #10218)

**What:** vm-alai-support's OS disk (30GB, `Standard_B2als_v2`) was filling up from Docker/containerd data (8 co-located services). CEO decision (Opcija 2, 2026-04-29) approved adding a dedicated managed data disk and moving the Docker data-root off the OS disk. Executed by FlowForge (Kelsey Hightower) 2026-04-30.

**Services affected (all on vm-alai-support):** bookstack, documenso, planka, baikal, vaultwarden, grafana, prometheus, caddy.

### 7.1 Disk Spec (tool-verified 2026-07-28 via `az disk show`)

| Field | Value |
|-------|-------|
| Disk name | `disk-alai-support-data` |
| Resource group | `rg-alai-support` (subscription `5b0b4d9b-e677-464e-abf0-5170cbce3b8e`) |
| Region | swedencentral |
| Size | 128 GB |
| SKU | **StandardSSD_LRS** (Standard SSD, tier "Standard") |
| Mount | `/mnt/data` (`/dev/sda1`, ext4) |
| Attached to | vm-alai-support |
| Created | 2026-04-30T03:33:33Z |
| Docker daemon `data-root` | `/mnt/data/docker` (`/etc/docker/daemon.json`) |

**⚠️ Discrepancy vs. task record:** MC #10218's title and DoD evidence describe this as "Premium SSD P10 128GB / Premium_LRS." Live Azure state (checked 2026-07-28) shows the actual disk is **StandardSSD_LRS** (Standard SSD "E10" 128GB tier), not Premium. Cost/perf implications: Standard SSD is cheaper and lower-IOPS than Premium P10 — if a Premium disk was genuinely intended, this is an open gap; if Standard was chosen deliberately for cost, the task title/evidence text was simply inaccurate. Not re-litigated here — flagging so nobody cites "Premium P10" as fact downstream.

**Mechanism note:** the task title says "update docker-compose.yml bind paths," but `/opt/alai/docker-compose.yml` uses named Docker volumes (e.g. `bookstack_config`, `caddy_data`), not host bind paths. The actual migration mechanism was repointing the Docker daemon's `data-root` to `/mnt/data/docker` in `/etc/docker/daemon.json` — compose files were not edited for path changes.

### 7.2 Pre-requisite (verified before migration)

Azure Backup (RSV) first recovery point confirmed to exist before any migration step ran (MC #10210):
- Recovery point ID `7702054437221953196`, 2026-04-29 22:09Z, `FileSystemConsistent`.
- Vault: `rsv-alai-support`, container `vm-alai-support`.

### 7.3 Migration Steps (as executed)

1. Confirm RSV recovery point exists (#10210, above).
2. `az disk create --name disk-alai-support-data -g rg-alai-support ...` (128GB).
3. Attach disk to vm-alai-support (LUN0 per DoD evidence).
4. SSH to VM, format new disk ext4, mount at `/mnt/data`.
5. Stop Docker/containerd.
6. `rsync` `/var/lib/docker` → `/mnt/data/docker` and `/var/lib/containerd` → `/mnt/data/containerd`.
7. Set `"data-root": "/mnt/data/docker"` in `/etc/docker/daemon.json`.
8. Start Docker/containerd; verify all 8 services (20 containers) come up healthy.
9. Log phase completion: `~/system/specs/system-reform-status.log` (`DATA-DISK-MIGRATION | STARTED | MC:#10218`).
10. `mc.js ready 10218` with DoD evidence; task marked done 2026-04-30 10:13:51.

**Result (DoD evidence, MC #10218):** 20 containers healthy post-migration. Old `/var/lib/docker` and `/var/lib/containerd` left in place pending a follow-up cleanup task (zero-FD check required first).

### 7.4 Old Directory Cleanup (7-day deletion window, MC #10228)

- **Target:** `/var/lib/containerd` (16G) + `/var/lib/docker` (2.5G) = 18.5G to reclaim. OS disk was at 94% at time of ticket.
- **Gate:** zero open file descriptors on the old dirs confirmed before deletion (safety check — nothing still had them open post-migration).
- **Deadline:** 2026-05-07 (MC #10228 `Deadline` field). *(Task #10221's brief cites "~2026-05-06" as an estimate; the actual tracked deadline in MC is 2026-05-07.)*
- **Completed:** 2026-05-05 15:36:11 — 2 days ahead of deadline.

**Live verification (2026-07-28):** `/var/lib/docker` and `/var/lib/containerd` no longer exist on vm-alai-support — cleanup confirmed still in effect.

### 7.5 Current State (tool-verified 2026-07-28)

```
$ df -h / /mnt/data
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        29G   11G   19G  36% /
/dev/sda1       126G   19G  101G  16% /mnt/data

$ docker info | grep "Docker Root Dir"
Docker Root Dir: /mnt/data/docker

$ docker ps | wc -l
20
```

OS disk usage dropped from a pre-migration ~94% down to 36% (11G/29G) after the data-root move + old-dir cleanup. 20 containers running, matching the post-migration DoD count.

### 7.6 Rollback Procedure

No pre-written rollback script exists for this migration; the safety net during the 7-day window (before #10228 cleanup) was the untouched copy of `/var/lib/docker` and `/var/lib/containerd` on the OS disk plus the confirmed Azure Backup RSV recovery point (Section 7.2). **After cleanup (2026-05-05), the OS-disk copies no longer exist** — rollback today means either restoring from the Azure Backup RSV recovery point, or reversing the data-root pointer if `/mnt/data` itself is the failure:

**A. Data disk corrupted / unmountable (VM boots, OS disk fine):**
```bash
ssh -i ~/.ssh/azure_alai alai-admin@4.223.110.181
sudo systemctl stop docker
# point data-root back to local OS disk (temporary, degraded — no persisted volumes)
sudo sed -i 's#/mnt/data/docker#/var/lib/docker#' /etc/docker/daemon.json
sudo systemctl start docker
# Services will start with EMPTY volumes — this is a stop-gap to get containers running,
# not a data recovery step. Follow with Option B to recover actual data.
```

**B. Full VM/data restore (disk lost, or data corruption needing point-in-time recovery):**
```bash
# Restore from Azure Backup RSV (rsv-alai-support) to the recovery point closest
# to the desired time, or a later automatic RP if scheduled backups exist post-2026-04-29.
az backup restore restore-disks \
  --resource-group rg-alai-support --vault-name rsv-alai-support \
  --container-name "IaasVMContainer;iaasvmcontainerv2;rg-alai-support;vm-alai-support" \
  --item-name "VM;iaasvmcontainerv2;rg-alai-support;vm-alai-support" \
  --rp-name <recovery-point-name> --storage-account <staging-storage-account>
# Then attach recovered data disk to vm-alai-support, verify /mnt/data mount and
# docker data-root, restart services, verify all 8 services healthy (curl each domain).
```

**Verify after any rollback:**
```bash
curl -sI https://docs.alai.no | head -1   # BookStack
curl -sI https://vault.alai.no | head -1  # Vaultwarden
curl -sI https://sign.alai.no | head -1   # Documenso
curl -sI https://grafana.alai.no | head -1
curl -sI https://boards.alai.no | head -1 # Planka
```

---

## 8. Escalation

| Issue | L1 Action | L2 Escalation | L3 Escalation |
|-------|-----------|---------------|---------------|
| Deploy failure | Review build logs; check package.json/next.config.js | Kelsey investigates Cloudflare Pages logs | Contact Cloudflare support via dashboard |
| 5xx errors (< 5 min) | Execute rollback (Section 2) | Kelsey reviews last commit for breaking change | CEO notification + DR activation (Section 6) |
| SSL cert not renewing | Verify DNS (Section 3) | Kelsey triggers manual renewal or contacts CF support | Switch to Let's Encrypt via Azure VM |
| SENTINEL false alerts | Verify site UP via curl; adjust timeout | Kelsey reviews SENTINEL script logic | Disable SENTINEL for that site; use external monitor |
| DNS not resolving | Verify Cloudflare DNS records; check registrar NS | Kelsey checks registrar portal for NS change | Contact registrar support |

**Key Contacts:**
- L2: Kelsey Hightower (FlowForge agent) via MC task
- L3: CEO (Alem Basic) via Slack DM or phone (+47 404 74 251)

---

## 9. Maintenance Schedule

| Task | Frequency | Owner | How |
|------|-----------|-------|-----|
| Test rollback procedure | Monthly | Proveo (Angie Jones) | Execute rollback on staging site; verify < 60s |
| Review SENTINEL alerts | Weekly | Kelsey | Check Slack `#infra-alerts` for false positives |
| Update dependency versions | Weekly | Renovate bot | Auto-merge minor/patch; manual review major |
| Backup DNS zone config | Weekly | Automated cron | Exports to `~/system/backups/dns/` |
| Verify SSL certs valid | Daily | SENTINEL | Auto-alert if < 30 days to expiry |

---

## 10. Related Docs

- [ALAI Static Hosting Blueprint](https://docs.alai.no) — Architecture & migration plan
- [Infrastructure Inventory](~/system/docs/infrastructure-inventory.md) — All ALAI sites & services
- [SENTINEL Reliability Sprint](~/system/docs/runbooks/sentinel-reliability.md) — Monitoring architecture
- [Incident Response Playbook](~/system/docs/runbooks/incident-response-playbook.md) — General incident workflow

---

## 11. Change Log

| Date | Change | Author |
|------|--------|--------|
| 2026-04-20 | Initial version — rollback, SSL, migration, SENTINEL | Skillforge (MC #8491) |
| 2026-07-28 | Added Section 7 (Azure VM Data Disk Migration — vm-alai-support, MC #10218); flagged Premium_LRS vs. live StandardSSD_LRS discrepancy; renumbered Sections 8–11 | Skillforge (MC #10221) |
| 2026-07-28 | Section 2 Step 2: replaced deprecated `wrangler pages deployment rollback` (removed in wrangler 4.x) with CF API re-deploy path, matching the blueprint fix already applied under MC #8494 | FlowForge (MC #8494) |