# Azure LightRAG Migration — Complete Runbook

# Azure LightRAG Migration — Complete Runbook

> **CURRENT RUNTIME — AZURE ARCHITECTURE SUPERSEDED (MC #106747; G4 offline refresh MC #107082):** LightRAG now runs on the local Mac Studio Docker stack. The last qualified read-only runtime evidence confirms `vm-alai-lightrag` is `VM deallocated` and the local `lightrag`/`lightrag-neo4j` topology is authoritative under `/Users/makinja/system/lightrag-local/docker-compose.local.yml`. MC #107082 prepared only an offline `d38f399...` client / `508e618...` server G4 bundle; it created no authority receipt, UTC window, deploy, restart, or runtime effect. The Azure instructions below remain historical architecture and rollback context and are intentionally reported as documentation drift, not current execution guidance. See MC #106747/#106767, `/Users/makinja/system/lightrag-local/mc106747-migration-note.md`, and `rag-queue-recovery-106978.md`.
>
> **Domain note (2026-05-17):** This doc was written when hostnames were `lightrag.basicconsulting.no` and `ollama.basicconsulting.no`. Both have since migrated to `lightrag.alai.no` and `ollama.alai.no`. Historical command examples below retain original hostnames for accuracy; use alai.no equivalents in live ops.

**Status:** COMPLETED 2026-04-18  
**Team Lead:** Kelsey Hightower (FlowForge)  
**Architect:** Petter Graff (CodeCraft)  
**Data Lead:** Martin Kleppmann (CodeCraft)  
**Validator:** Angie Jones (Proveo)  
**Documentation:** Skillforge  

## Operational Update — 2026-05-20

MC #101607 repaired a query-path regression after Azure LightRAG health was green but `/query` returned HTTP 500 because `LLM_MODEL=qwen3:8b-q8_0` was not available behind the Ollama tunnel. Azure `/home/alai-admin/lightrag/.env` now uses `LLM_MODEL=llama3.1:8b`, and the `lightrag` container was force-recreated without deleting volumes. Direct Azure endpoint `http://20.240.61.67:9621` verified: `/health` healthy and `/query` HTTP 200. Local Anvil/Pi mock config currently uses the Azure direct URL as `lightrag.base_url`.

MC #101611 added client-side Cloudflare Access service-token support for the LightRAG wrapper and key consumers. When `lightrag.base_url=https://lightrag.alai.no`, provide `LIGHTRAG_CF_ACCESS_CLIENT_ID` and `LIGHTRAG_CF_ACCESS_CLIENT_SECRET` (or generic `CF_ACCESS_CLIENT_ID` / `CF_ACCESS_CLIENT_SECRET`). Do not commit secrets. Do not switch canonical `lightrag.base_url` from Azure direct to the public URL until a real service token validates `/health` and `/query` through Cloudflare Access.

Evidence: `/tmp/verify-101607/SUMMARY.md`, `/tmp/verify-101609/SUMMARY.md`, `/tmp/verify-101611/SUMMARY.md`.

## Operational Update — 2026-07-28 (MC #101501)

`~/system/tools/lightrag-bulk-upload.js` ("the pump") default `--rate` increased from 10 to 25 docs/min. No launchd/cron caller passes `--rate` explicitly, so the hardcoded default (`main()`, previously `parseInt(getArg('--rate') || '10')`) was the effective live rate limit — it now reads `|| '25'`. Doc-comment usage examples at the top of the file updated to match. Evidence: `/tmp/evidence-101501/verification.txt` (node -c syntax check + git diff).

## CRITICAL Security Update — 2026-06-18/19

**MC #103912:** Public exposure incident and migration to app-layer JWT authentication.

### Root Cause (2026-06-18)

Boot.sh/discover.js reported "LightRAG DOWN" (false negative). NSG `vm-alai-lightragNSG` only allows port 9621 from `46.46.240.0/20` + Cloudflare ranges; orchestrator host egress IP (`92.221.168.61`) was not whitelisted. LightRAG container was healthy throughout.

### Security Incident Timeline

- **18:19 UTC:** New Cloudflare tunnel `lightrag-alai` (c79ffe4d-0be9-40c6-9eaa-608edeb307db) created on Azure VM to resolve access issue. DNS `lightrag.alai.no` updated to point to tunnel.
- **18:19–18:38 UTC (~19 min):** **PUBLIC EXPOSURE.** Tunnel published `lightrag.alai.no` with NO enforced authentication. Full knowledge graph (15,840 docs incl. internal company/client/partner names) queryable from open internet.
- **18:38 UTC:** Tunnel stopped after external-vantage verification confirmed HTTP 200 leak (builder's "302 BLOCKED" claim was tested from bypass-whitelisted IP and invalid).

**Why Cloudflare Access Failed:**

Attempted 3 times:
1. Wrong AUD tag (used app ID instead of correct AUD `45433679774e5bb11a3a5c284cf3a71e9f8865c93c513f5cc204e382e96cff8d`)
2. Wrong team name (`alai` instead of `alai-no`)
3. `originRequest.access` enforces user JWT tokens (browser login), NOT service token headers

**KEY LESSON:** Verify allowlist/auth ONLY from a vantage NOT in any bypass list. CF Access IP-Bypass policy whitelists `92.221.168.61`; testing from that IP always bypassed Access regardless of config.

### Final Solution: App-Layer JWT Authentication

**Enforcement Point:** LightRAG FastAPI application (sbnb/lightrag container), NOT Cloudflare edge.

**Configuration:**

VM path: `/home/alai-admin/lightrag/.env`

```bash
AUTH_ACCOUNTS=alai-system:<password_hash>
TOKEN_SECRET=<jwt_secret>
WHITELIST_PATHS=/health
```

Credentials: Vaultwarden item `67dc69b5-b1cb-4892-970e-b4d60380378f` (LightRAG API Auth).

**Auth Flow:**

1. Client POSTs to `/login` with username + password (form data)
2. Server validates credentials, returns JWT (48h expiry)
3. Client includes `Authorization: Bearer <token>` on subsequent requests
4. `/health` is PUBLIC (whitelisted for boot.sh probes)
5. `/query`, `/insert`, `/api/*` require valid JWT (HTTP 401 without auth)

**Verification (external-vantage required):**

```bash
# Unauthenticated MUST return 401
curl -i -X POST https://lightrag.alai.no/query \
  -H 'Content-Type: application/json' -d '{"query":"test"}'
# Expected: HTTP/1.1 401 Unauthorized

# Health endpoint PUBLIC
curl -s https://lightrag.alai.no/health | jq -r '.status'
# Expected: "healthy"

# Authenticated access
TOKEN=$(curl -s -X POST 'https://lightrag.alai.no/login' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d 'username=alai-system&password=<from_vaultwarden>' \
  | jq -r '.access_token')

curl -s -X POST https://lightrag.alai.no/query \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"query":"test","mode":"naive","top_k":1}' | jq -r '.response'
# Expected: HTTP 200 + query results
```

**Architecture:**

```
External Client (unauthenticated)
  ↓ HTTPS POST /query
Cloudflare (lightrag.alai.no)
  ↓ Tunnel (c79ffe4d)
Azure VM cloudflared
  ↓ HTTP localhost:9621
LightRAG FastAPI Server
  → JWT Validator Middleware
  → HTTP 401 Unauthorized ❌ BLOCKED

External Client (authenticated with JWT)
  ↓ HTTPS POST /query + Authorization: Bearer <jwt>
Cloudflare → Tunnel → VM cloudflared → localhost:9621
LightRAG FastAPI Server
  → JWT validates (TOKEN_SECRET, 48h expiry)
  → HTTP 200 + query results ✅ ALLOWED
```

### Known Residuals

- Vestigial CF Access Application (c62b46b1-43f4-4967-9b99-cabfefb6b99b) + IP-Bypass policy still exist but not enforced (app-layer takes precedence).
- `/health` endpoint exposes doc counts + config (ollama host, storage backends) publicly. Accepted for health checks.
- The AUTH_ACCOUNTS password appeared in plaintext in agent reports/evidence during initial fix; recommend rotation as hygiene.

### Rotating LightRAG Credentials

```bash
# Generate new password hash
NEW_PASS=$(openssl rand -hex 32)
HASH=$(echo -n "$NEW_PASS" | sha256sum | awk '{print $1}')

# Update on VM
ssh -i ~/.ssh/azure_alai alai-admin@20.240.61.67
cd ~/lightrag
# Edit .env: AUTH_ACCOUNTS=alai-system:<new_hash>
docker compose restart lightrag

# Test
curl -s -X POST 'https://lightrag.alai.no/login' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d "username=alai-system&password=$NEW_PASS" | jq -r '.access_token'

# Update Vaultwarden item 67dc69b5-b1cb-4892-970e-b4d60380378f with new password
```

### boot.sh / discover.js Integration

Config: `~/system/tools/alai-config-mock.json`

```json
{
  "lightrag.base_url": "https://lightrag.alai.no"
}
```

Credentials loaded from environment (NOT committed to git):

```bash
export LIGHTRAG_USERNAME="alai-system"
export LIGHTRAG_PASSWORD="<from_vaultwarden>"
```

boot.sh/discover.js authenticate on first request, cache JWT for 48h, refresh when expired.

Evidence: `/tmp/evidence-103912/app-layer-auth-proof.md`, `/tmp/evidence-103912/verification-final.md`

---

## Executive Summary

**Why migrated:** Docker Desktop failed 3 times on 2026-04-18, causing LightRAG outages impacting all ALAI knowledge operations (discover.js, autocoder.js, retrieval-orchestrator.js). Local dependency became unacceptable single point of failure.

**What changed:**
- LightRAG + Neo4j moved from Mac Studio Docker Desktop → Azure VM `vm-alai-lightrag` (swedencentral)
- Ollama remains on Mac Studio but exposed via Cloudflare tunnel with Zero Trust IP whitelist
- 8 consumer files updated to use `https://lightrag.basicconsulting.no` instead of `http://localhost:9621`
- Data migrated: 497MB snapshot from Azure Blob backup 20260418-085317 (zero data loss)

**Result:**
- ✅ Docker Desktop crashes no longer affect LightRAG availability
- ✅ Query latency acceptable: p50 ~2-3s vs ~1s local (30-60ms tunnel overhead + network)
- ✅ Rollback capability preserved: 5-15 min return to local if needed
- ✅ Azure cost: ~$30/month (B2s_v2 VM), pulled from credits

---

## Architecture Diagram

```mermaid
sequenceDiagram
    participant Consumer as Mac Studio Consumer<br/>(discover.js, autocoder.js, etc.)
    participant CF1 as Cloudflare<br/>lightrag.basicconsulting.no
    participant Tunnel1 as Mac Studio<br/>cloudflared tunnel
    participant VM as Azure VM<br/>20.240.61.67:9621<br/>LightRAG container
    participant CF2 as Cloudflare<br/>ollama.basicconsulting.no
    participant Tunnel2 as Mac Studio<br/>cloudflared tunnel
    participant Ollama as FORGE<br/>10.0.0.2:11434<br/>Ollama service

    Consumer->>CF1: HTTPS query request
    CF1->>Tunnel1: Route via tunnel
    Tunnel1->>VM: Forward to Azure VM:9621
    VM->>CF2: HTTPS request for LLM inference
    CF2->>Tunnel2: Route via tunnel (Zero Trust IP check)
    Tunnel2->>Ollama: Forward to FORGE:11434
    Ollama-->>Tunnel2: Inference response
    Tunnel2-->>CF2: Return
    CF2-->>VM: LLM result
    VM->>VM: Build knowledge graph (Neo4j)
    VM-->>Tunnel1: Query response
    Tunnel1-->>CF1: Return
    CF1-->>Consumer: HTTPS response
```

**Key insight:** Cloud LightRAG talks back to on-prem Ollama via second tunnel. Both services behind Cloudflare Zero Trust.

---

## Resources Created

### Azure Resource Group
- **Name:** `rg-alai-lightrag`
- **Location:** swedencentral
- **Purpose:** Dedicated to LightRAG stack

### Virtual Machine
- **Name:** `vm-alai-lightrag`
- **Size:** Standard_B2s_v2 (2 vCPU, 8 GB RAM, Intel x86_64)
- **OS:** Ubuntu 22.04 LTS
- **Public IP:** 20.240.61.67 (static)
- **OS Disk:** 30GB Premium SSD
- **Data Disk:** Not used (Docker volumes on OS disk sufficient for now)
- **SSH:** `ssh -i ~/.ssh/azure_alai alai-admin@20.240.61.67`
- **Docker:** 29.4.0
- **Docker Compose:** 2.30.3

### Network Security Group (NSG)
**Name:** `vm-alai-lightragNSG`

| Rule Name | Priority | Direction | Port | Source | Purpose |
|-----------|----------|-----------|------|--------|---------|
| `default-allow-ssh` | 1000 | Inbound | 22 | 46.46.251.40/32 | SSH from Mac Studio ISP |
| `allow-lightrag-macstudio` | 100 | Inbound | 9621 | 46.46.251.40/32 | Direct access (backup) |
| `allow-cloudflare-lightrag` | 110 | Inbound | 9621 | Cloudflare IP ranges | Tunnel ingress |

**Important:** Mac Studio ISP IP (46.46.251.40) is residential and may rotate. When rotation happens, SSH and direct API access will fail. Update NSG rules accordingly (see Troubleshooting).

### Cloudflare DNS Records
- **lightrag.basicconsulting.no** → CNAME to `3315a609-7934-45c5-ad0c-56d86d16374d.cfargotunnel.com`
  - Tunnel config: `service: http://20.240.61.67:9621`
  - Routes external consumers to Azure VM via Mac Studio tunnel (relay)
- **ollama.basicconsulting.no** → CNAME to `3315a609-7934-45c5-ad0c-56d86d16374d.cfargotunnel.com`
  - Tunnel config: `service: http://10.0.0.2:11434`
  - Routes Azure VM back to FORGE Ollama via Mac Studio tunnel
  - Zero Trust policy: IP whitelist includes Azure VM egress IP + Mac Studio

---

## Data Migration

### Source
- **Azure Blob Storage:** `plockfrontstaging/lightrag-backup/20260418-085317/`
- **Size:** 497MB compressed (4 tarballs + manifest + README)
- **Volumes:**
  - `lightrag-data.tar.gz` (312MB) — KV store + inputs
  - `lightrag-kg.tar.gz` — Knowledge graph files
  - `lightrag-cache.tar.gz` — LLM response cache
  - `lightrag-neo4j-data.tar.gz` (169MB) — Neo4j graph entities + relations

### Restore Process
1. Download from Azure Blob to VM `/tmp/restore/`
2. `shasum -a 256 -c MANIFEST.sha256` — verified all 4 tarballs
3. Created 4 Docker volumes
4. Extracted each tarball into its volume using Alpine container
5. Started LightRAG + Neo4j containers
6. Verified entity count in Neo4j matched pre-migration snapshot

**Data loss:** ZERO. Snapshot taken immediately before migration.

---

## Consumer Files Cut Over

8 files updated from `http://localhost:9621` → `https://lightrag.basicconsulting.no`:

| File | Purpose |
|------|---------|
| `~/system/tools/discover.js` | Universal search (tools, agents, docs, RAG) |
| `~/system/tools/lightrag.js` | LightRAG client wrapper |
| `~/system/tools/autocoder.js` | Code generation with RAG context |
| `~/system/tools/lightrag-bulk-upload.js` | Batch document ingestion |
| `~/system/tools/lightrag-migrate.js` | Migration utility |
| `~/system/tools/lightrag-outbox-ingest.js` | Outbox processor |
| `~/system/tools/retrieval-orchestrator.js` | Multi-source retrieval coordinator |
| `~/system/tools/system-regression.sh` | Health check suite |

**Pre-cutover backups:** Not created (files tracked in Git, easy revert via `git restore`).

---

## Operational Procedures

### Daily Health Check
```bash
# From Mac Studio
curl https://lightrag.basicconsulting.no/health

# Expected response:
# {"status":"healthy","working_directory":"/app/data", ...}
```

### SSH to VM
```bash
ssh -i ~/.ssh/azure_alai alai-admin@20.240.61.67
```

### Docker Container Management
```bash
# On Azure VM
cd ~/lightrag
docker compose ps              # Check status
docker compose logs -f         # Tail logs
docker compose restart         # Restart services
docker compose down && docker compose up -d  # Full restart
```

### Health Check from VM (tests Ollama tunnel)
```bash
# On Azure VM
curl -s https://ollama.basicconsulting.no/api/tags | jq '.models | length'
# Should return model count (e.g., 12)
```

### Azure Cost Check
```bash
# From Mac Studio
az consumption usage list --start-date $(date -u -v-7d +%Y-%m-%d) --end-date $(date -u +%Y-%m-%d) -o table
```

### Stop LightRAG (for maintenance)
```bash
# On Azure VM
cd ~/lightrag
docker compose stop
# Restart after maintenance
docker compose start
```

---

## Rollback Procedure — CRITICAL

**When to rollback:**
- Azure VM becomes unstable or unresponsive for >30 min
- Tunnel failures persist despite troubleshooting
- Cost overrun detected
- Any catastrophic issue requiring immediate local restore

**Expected rollback time:** 5-15 minutes  
**Data loss risk:** ZERO (local volumes preserved 7 days post-cutover)

### Step 1: Revert Consumer URLs
```bash
# On Mac Studio
cd ~/system/tools
for file in discover.js lightrag.js autocoder.js lightrag-bulk-upload.js \
            lightrag-migrate.js lightrag-outbox-ingest.js \
            retrieval-orchestrator.js system-regression.sh; do
  sed -i '' 's|https://lightrag.basicconsulting.no|http://localhost:9621|g' "$file"
done

# Verify
grep -l "localhost:9621" *.js *.sh
# Should list all 8 files
```

### Step 2: Restart Local Docker LightRAG
```bash
cd ~/system/docker/lightrag
docker compose up -d

# Wait for healthy status (30-60s)
docker compose ps
```

### Step 3: Verify Local Service
```bash
curl http://localhost:9621/health
# Expected: {"status":"healthy", ...}

# Run regression suite
bash ~/system/tools/system-regression.sh
# LightRAG checks should PASS
```

### Step 4: Deprovision Azure VM (optional, when convenient)
```bash
az group delete --name rg-alai-lightrag --yes --no-wait
# Deletes VM, NSG, disks, public IP
# Cloudflare DNS records remain (harmless)
```

**Post-rollback actions:**
1. Update `~/system/docs/runbooks/lightrag-backup.md` to revert to local backup flow
2. Notify Alem in Slack #ops-alai
3. Create MC task for post-mortem

---

## Troubleshooting

### Issue: "model not found" errors in LightRAG logs

**Cause:** Cloudflare tunnel `ollama.basicconsulting.no` routing to wrong backend.

**Diagnosis:**
```bash
# On Azure VM
curl -s https://ollama.basicconsulting.no/api/tags | jq '.models[].name'
# Should list qwen2.5-coder:32b-instruct-q8_0, bge-m3:latest, etc.
```

**Fix:**
1. Check Mac Studio tunnel config: `cat ~/.cloudflared/config.yml | grep -A2 ollama`
2. Should be `service: http://10.0.0.2:11434` (FORGE), NOT `http://localhost:11434` (ANVIL)
3. If wrong: edit config, restart tunnel: `launchctl kickstart -k gui/$(id -u)/com.john.cloudflared`

**Related incident:** 2026-04-18 mid-migration — tunnel initially pointed to ANVIL, causing q8_0 model not found. Changed to FORGE, resolved immediately.

---

### Issue: SSH connection timeout

**Cause:** Mac Studio ISP rotated IP; NSG rule `default-allow-ssh` still has old IP (46.46.251.40/32).

**Diagnosis:**
```bash
# On Mac Studio
curl https://ifconfig.co
# Compare to NSG rule source IP
az network nsg rule show -g rg-alai-lightrag --nsg-name vm-alai-lightragNSG -n default-allow-ssh --query "sourceAddressPrefix"
```

**Fix:**
```bash
NEW_IP=$(curl -s https://ifconfig.co)
az network nsg rule update \
  -g rg-alai-lightrag \
  --nsg-name vm-alai-lightragNSG \
  -n default-allow-ssh \
  --source-address-prefixes "${NEW_IP}/32"

# Verify
ssh -i ~/.ssh/azure_alai alai-admin@20.240.61.67
```

**Note:** Use `ifconfig.co` or `icanhazip.com`, NOT `ifconfig.me` (returns CDN IP on some networks).

---

### Issue: "connection refused" from consumer scripts

**Cause:** Cloudflare tunnel down or misconfigured.

**Diagnosis:**
```bash
# From Mac Studio
curl https://lightrag.basicconsulting.no/health
# If timeout/connection refused, tunnel is down

# Check tunnel process
ps aux | grep cloudflared
# Should show cloudflared running with config ~/.cloudflared/config.yml

# Check tunnel logs
tail -f ~/Library/Logs/cloudflared/cloudflared.log
```

**Fix:**
```bash
# Restart tunnel
launchctl kickstart -k gui/$(id -u)/com.john.cloudflared

# Wait 10-15s, retry
curl https://lightrag.basicconsulting.no/health
```

---

### Issue: Slow query responses (>10s p95)

**Cause 1:** Network path issue (Mac Studio → Cloudflare → Azure → Cloudflare → Mac Studio).

**Diagnosis:**
```bash
# On Azure VM
time curl -s https://ollama.basicconsulting.no/api/tags > /dev/null
# Should be <100ms

# From Mac Studio
time curl -s https://lightrag.basicconsulting.no/health > /dev/null
# Should be <200ms
```

**Cause 2:** Ollama FORGE overloaded (other tasks using models).

**Diagnosis:**
```bash
# On Mac Studio
curl http://10.0.0.2:11434/api/ps
# Check running models
```

**Fix:** Identify and throttle/stop competing workloads on FORGE.

---

### Issue: Neo4j "unable to allocate memory"

**Cause:** B2s_v2 has 8GB RAM; Neo4j + LightRAG + OS overhead can approach limit.

**Diagnosis:**
```bash
# On Azure VM
docker stats --no-stream
# Check memory usage percentages

free -h
# Check system memory
```

**Fix (short-term):**
```bash
# Restart containers to clear caches
cd ~/lightrag
docker compose restart
```

**Fix (long-term):** Upgrade to Standard_B2s_v2 → Standard_B4ms (4 vCPU, 16GB RAM, ~$60/month).

---

## Cross-References

- **Ollama Tunnel:** [ollama-cloudflare-tunnel.md](./ollama-cloudflare-tunnel.md) — tunnel config, Zero Trust policy, failure modes
- **Backup Flow:** [lightrag-azure-backup.md](./lightrag-azure-backup.md) — updated for cloud-resident LightRAG
- **Migration Plan:** `~/system/specs/lightrag-azure-migration-plan.md` — full task breakdown (14 tasks, 5 phases)

---

## Validation Evidence

**Completed by:** Angie Jones (Proveo), 2026-04-18

- ✅ Full round-trip: `discover.js "lightrag"` returns hits from cloud graph
- ✅ Ingest test: new document ingested via `lightrag-outbox-ingest.js`, confirmed in Neo4j
- ✅ Docker Desktop killed for 10 min: all ALAI functions remained operational
- ✅ Latency test: 20 queries — p50=2.3s, p95=7.1s (acceptable vs local p50=1.1s, p95=3.2s)
- ✅ Azure cost projection: 24h run = $1.20 (prorated to ~$36/month; budget set at $30/month, within tolerance)
- ✅ First Azure-native backup: 20260418-203045 snapshot uploaded to Blob (512MB)

**Evidence bundle:** `~/system/evidence/azure-lightrag-migration-20260418/SUMMARY.md`

---

## Next Steps

1. **Monitor for 7 days** — track latency, cost, uptime. If stable, delete local Docker volumes.
2. **Update backup flow** — migrate launchd plist to SSH-based Azure VM snapshot (see lightrag-azure-backup.md).
3. **Consider FORGE failover** — expose FORGE Ollama via second tunnel endpoint for redundancy.
4. **Auto-scaling evaluation** — if query volume grows, consider Azure Container Instances or AKS migration.

---

**Document Owner:** Skillforge  
**Last Updated:** 2026-04-18  
**Approved By:** Petter Graff (Architecture), Kelsey Hightower (Infra), Alem Basic (CEO)