# LightRAG Recovery — Session 6c3cc7f0 Post-Mortem (2026-06-29)

# LightRAG Recovery — Session 6c3cc7f0 Reconciliation

> **DocType:** Incident Post-Mortem + Reconciliation
> **Session:** `6c3cc7f0-ff37-46c3-99ab-952b75a451b8` (2026-06-29)
> **Documented by:** FlowForge reconciliation pass (2026-06-30)
> **MC Task:** [#104528 LightRAG drain + FORGE migration eval](https://boards.alai.no)
> **Incident Memo:** `incident_lightrag_403_extraction_and_chunkvdb_corruption_2026-06-29.md`

---

## 1. Context: The "Do Not Touch" Rule

MEMORY.md contains the standing rule:

> "LightRAG FIXED not dead — never re-diagnose/touch neo4j (project_lightrag_fixed_2026-06-12)."

Session 6c3cc7f0 **did** run approximately 26 `az vm run-command invoke` calls against `vm-alai-lightrag` (RG-ALAI-LIGHTRAG). See [Flag A](#flag-a) below for the alignment assessment.

---

## 2. Initial State (verified from transcript)

| Parameter | Value |
|-----------|-------|
| VM | `vm-alai-lightrag`, RG-ALAI-LIGHTRAG, public IP `20.240.61.67` |
| Compose dir | `/home/alai-admin/lightrag/` |
| LightRAG version | api_version: 0154, core: 1.3.4 |
| `LLM_BINDING_HOST` (pre-session) | `https://ollama.basicconsulting.no` |
| `EMBEDDING_BINDING_HOST` (pre-session) | `http://ollama:11434` (correct, already internal) |
| `LLM_MODEL` | `llama3.1:8b` |
| Internal ollama models | `llama3.2:1b`, `bge-m3:latest` — llama3.1:8b was NOT present |
| processed | 19,236 |
| pending | 40,559 |
| failed | 8–10 |
| pipeline_busy | false (stuck; pipeline not retrying) |

---

## 3. Root Cause Chain (Three Links)

The session established the following confirmed chain of failures, documented in the incident memo:

### Link 1 — Stale LLM_BINDING_HOST

`.env` line 10 pointed to `https://ollama.basicconsulting.no`, a leftover from before the domain migration to `alai.no` (pre-2026-05-17). The compose default in `docker-compose.yml` line 39 was `${LLM_BINDING_HOST:-https://ollama.basicconsulting.no}`.

### Link 2 — Dead Tunnel Origin

`ollama.alai.no` is served by cloudflared tunnel **"mattermost"** (id `3315a609-7934-45c5-ad0c-56d86d16374d`). Its origin rule was pointing to `http://10.0.0.2:11434` (FORGE Ollama). FORGE switched from Ollama to MLX (port 11435) on 2026-06-25. The Ollama :11434 service on FORGE was dead. The tunnel needed repointing to `http://localhost:11434` (ANVIL local Ollama, which does have llama3.1:8b + bge-m3).

### Link 3 — Ollama DNS-Rebind Guard (the actual 403 killer)

Ollama returns HTTP 403 to any request whose `Host` header is not `localhost` or `127.0.0.1`. cloudflared by default forwards the original `Host: ollama.alai.no` header to the origin. This means every LightRAG extraction call received a 403 from ollama regardless of CF Access config.

**Proof in transcript:** `curl -H 'Host: ollama.alai.no' localhost:11434/api/tags` = 403 vs `curl -H 'Host: localhost' localhost:11434/api/tags` = 200.

**Lesson (durable):** `cf-cache-status: DYNAMIC` on a 403 response means the origin returned it — Cloudflare Access was NOT the gating layer. Do not assume `403 content-length:0 server:cloudflare` means Access blocked it.

---

## 4. What the Session Did — Chronological

### Step 1: Diagnosis

Ran `az vm run-command invoke` to read container state, `docker ps`, LightRAG logs, `.env` contents, and `docker exec lightrag env`. Confirmed extraction LLM host and 403 pattern.

### Step 2: Wrong Fix — Internal Ollama (MY MISTAKE #1)

Pulled `llama3.1:8b` to the internal ollama container (`docker exec ollama ollama pull llama3.1:8b`, 4.9 GB). Edited `.env` to set `LLM_BINDING_HOST=http://ollama:11434`. Ran `docker compose up -d lightrag`.

**Why this was wrong:** The VM is 2 vCPU / 7.8 GB RAM / no GPU. This architecture is documented in `project_lightrag_topology_fix_2026-06-03` — the LLM is remote BY DESIGN because the VM cannot host an 8B model. The result was OOM-thrash: ollama logged "llm server loading model" in a loop, never ready, ~195 MB RAM remaining. Zero docs drained.

### Step 3: vdb_chunks.json Corruption (MY MISTAKE #2)

Ran `docker compose up -d lightrag` to recreate the container with the new `.env` (without a graceful stop first). The default stop timeout (10 seconds) sent SIGKILL mid-flush.

**Effect:** `vdb_chunks.json` was truncated from 132 MB to 48 MB. The matrix (chunk embeddings) was lost. On next start, `nano_vectordb.load_storage` raised `JSONDecodeError` → container entered crash-loop.

The large vdbs (`vdb_entities.json` 516 MB, `vdb_relationships.json` 759 MB) survived only by write-order luck.

**Lesson (repeat of feedback_no_restart_prod_for_cosmetic_2026-06-04):**

> This container's graceful stop requires `-t 180` (measured flush time: 3 minutes 4 seconds).  
> Never use default-timeout restart or `docker compose up -d` without a prior `docker compose stop -t 180`.  
> Verify backup integrity before any container lifecycle operation on this VM.

### Step 4: Recovery from Backup

**Source:** `~/system/backups/lightrag/20260628-040003/lightrag-data.tar.gz` (weekly cron snapshot)

**Content:** Valid `vdb_chunks.json` — 132 MB, JSON valid, matrix present, 17,234 data entries, content dated Jun 7 (consistent with entity/relationship vdbs frozen at that date).

**Recovery sequence:**

1. Extract `vdb_chunks.json` locally from tarball.
2. Upload to Azure Blob: `plockfrontstaging/lightrag-backup/restore-20260629/vdb_chunks.json`.
3. Generate SAS URL (short TTL).
4. On VM via `az vm run-command`: `docker compose stop -t 180` (graceful, 3 min 4 sec measured).
5. On VM: `curl <SAS_URL>` into volume mount `/mnt/docker-data/docker/volumes/lightrag-data/_data/vdb_chunks.json`.
6. sha256 verify: `b24174…` (matched local extracted file).
7. Update `.env` to correct LLM host (see step 5 below).
8. `docker compose up -d`.

**Verified result:** `t=80s` RestartCount=0, Health=healthy. nano-vectordb loaded: `vdb_chunks.json 17234 data` — clean load, no JSONDecodeError. Available memory recovered from ~195 MB to 4.1 GB.

### Step 5: Correct Fix — Cloudflared Tunnel + httpHostHeader

**Critical infra fact (discovered in session):** The tunnel is **dashboard-managed (remote config)**. Editing `/home/alai-admin/cloudflared/config.yml` locally does nothing — cloudflared pulls remote config and logs "Updated to new configuration version=N". `cloudflared tunnel ingress rule <url>` reads the LOCAL file and misleads.

**Fix path:** CF API:
- Endpoint: `GET/PUT /accounts/d0ac2afb6bb5b298723b85a114151a04/cfd_tunnel/3315a609-7934-45c5-ad0c-56d86d16374d/configurations`
- Auth: Bitwarden item "Cloudflare Global API Key" (X-Auth-Email + X-Auth-Key). The generic "Cloudflare api token" in BW lacks tunnel/Access scope.
- Change: Repoint `ollama.alai.no` origin from `http://10.0.0.2:11434` → `http://localhost:11434`. Add `originRequest.httpHostHeader: "localhost:11434"` to bypass ollama's DNS-rebind guard.
- The tunnel serves 32 ingress rules; only the target rule was changed, then full config was PUT back.

**CF Access (Red Herring):** IP-bypass policy already included VM `20.240.61.67/32` in BOTH the exact `ollama.alai.no` app (id `bdc17e6a`) and the `*.alai.no` wildcard app (id `cd7cf0f0`). CF Access was NOT the gating layer.

**Verified after fix:** `VM → ollama.alai.no/api/tags` = 200. `/api/chat llama3.1:8b` = 200 "OK". LightRAG logs showed "Merge N/E entities+relations". Zero new 403 errors.

---

## 5. Health Counts at Session Handoff

Taken from final `docker exec lightrag python3` health probe in transcript:

| Metric | Value |
|--------|-------|
| `llm_binding_host` | `https://ollama.alai.no` |
| `embedding_binding_host` | `http://ollama:11434` |
| `llm_model` | `llama3.1:8b` |
| `processed` | 19,237 (incremented from 19,236 during session) |
| `pending` | ~40,563–40,564 |
| `failed` | 8 |
| `processing` | 2 |
| `pipeline_busy` | True |
| RestartCount | 0 |
| 403 errors post-fix | 0 |

Note: `processed` incremented by 1 (+180s observation) after the correct fix was applied, confirming the pipeline was draining.

---

## 6. MC #104528 — Blocker Detail

**Task:** LightRAG drain + FORGE migration eval  
**Status:** blocked (awaiting_forge — GOTCHA doc missing, /prompt-forge 104528 required before unblock)

**Stated blocker in MC description:** "add Cloudflare Access Bypass policy for VM egress IP `20.240.61.67/32` to `ollama.alai.no` app (Zero Trust, account `d0ac2afb6bb5b298723b85a114151a04`)"

**Reconciliation note:** The MC task description reflects the diagnosis made early in the session before the DNS-rebind guard was identified as the real root cause. Per the incident memo and transcript evidence, the CF Access bypass was already in place for `20.240.61.67`. The actual fix (httpHostHeader + tunnel repoint) was applied and verified working within the same session. The stated MC blocker may now be resolved. This should be re-verified before the next session touches #104528.

**Open sub-tracks in #104528:**
1. Evaluate migrating LightRAG + neo4j + ollama to FORGE (Mac Studio M3 Ultra 256 GB, OrbStack) to remove cloud/tunnel fragility. Prior reason to leave Mac: Docker Desktop crashed 3x on 2026-04-18. Counter: OrbStack removes that SPOF.
2. Build LightRAG idle-watchdog: re-trigger pipeline when `pending>0 && pipeline_busy=false` for >N minutes. Without this, any future CF/tunnel blip will re-stick the 40k backlog indefinitely.
3. Proveo validation task required per ZAKON PLAN before any drain claim.
4. Skillforge BookStack documentation — this page closes that gap.

---

## 7. Flags

### Flag A — "Do Not Touch" Rule Assessment {#flag-a}

The MEMORY.md rule states: "LightRAG FIXED not dead — never re-diagnose/touch neo4j (project_lightrag_fixed_2026-06-12)."

**Assessment:** The session's VM access was triggered by a real new incident (40k docs stuck pending, extraction 403 for an unknown period). The "FIXED" tag in MEMORY.md referred to a prior false-negative health-check (2026-06-12), not to permanent immunity from new incidents. The neo4j volumes were not touched. The touch was operationally justified.

**However:** The session made two compounding mistakes (internal-ollama wrong fix + non-graceful recreate) that would not have occurred if the topology documentation (`project_lightrag_topology_fix_2026-06-03`) had been read before acting. The "do not touch" rule exists partly to prevent exactly this class of well-intentioned damage. Standing rule recommendation: before any `az vm run-command` that modifies LightRAG container state, read the topology memo and the backup runbook first.

**Verdict:** Rule spirit violated (touch caused damage). Rule letter partially applies — the rule's "neo4j" qualifier was respected; the broader "do not re-diagnose" guidance was not.

### Flag B — CF Access Service Token Plaintext Exposure {#flag-b}

During transcript tool result output (a `Read` call on `lightrag.js` source), the following values were printed in plaintext:

```
"lightrag.cf_access_client_id": "4248b2c109e87e09faf3fb82a90eeafd.access"
"lightrag.cf_access_client_secret": "[REDACTED — do not reprint]"
```

These are CF Access service token credentials for `lightrag.alai.no`. They were embedded in an `alai_config` block in the source file and were printed verbatim in the session transcript (file `6c3cc7f0-ff37-46c3-99ab-952b75a451b8.jsonl`).

**Risk:** The transcript file is readable on-disk at `~/.claude/projects/-Users-makinja/`. Any process or agent with filesystem access can read these values.

**Recommended action:** Rotate the CF Access service token for `lightrag.alai.no` (generate a new CID/CSEC pair in Cloudflare Zero Trust → Access → Service Auth). Update `alai_config` / Bitwarden. Sub-agent prompts should explicitly prohibit echoing secrets in output.

---

## 8. .env Configuration State at Handoff

File: `/home/alai-admin/lightrag/.env`

| Key | Value |
|-----|-------|
| `LLM_BINDING_HOST` | `https://ollama.alai.no` |
| `LLM_MODEL` | `llama3.1:8b` |
| `EMBEDDING_BINDING_HOST` | `http://ollama:11434` |

Backups on VM: `.env.bak-pre-internal-llm-*` (the wrong internal-ollama fix), `.env.bak-pre-localembed-20260603` (older).

---

## 9. Architecture Notes (Durable)

- **VM compute:** 2 vCPU / 7.8 GB RAM / no GPU. Cannot host any >1B parameter LLM. Remote LLM is REQUIRED.
- **Tunnel is dashboard-managed.** Do not edit local `config.yml`. Use CF API for tunnel config changes.
- **Graceful stop requires `-t 180`.** Default 10s SIGKILL always corrupts in-flight vdb flushes.
- **Weekly backup cron** saves to `~/system/backups/lightrag/YYYYMMDD-HHMMSS/`. The Jun 28 backup was what saved this recovery. The Jun 7 content date of `vdb_chunks.json` confirms vdb vectors have not been updated since then (entities/relationships progressed further in memory but vdb chunk index was stale — re-embed from `kv_store_text_chunks.json` is a deferred open item).
- **`/health` doc_status_counts is the authoritative metric.** `lightrag.js status` returns false `Relations: 0` — do not use it for health assessment.

---

## 10. Related Runbooks

- `~/system/docs/runbooks/azure-lightrag-migration.md` — full migration history + security incident (2026-06-18)
- `~/system/docs/runbooks/lightrag-backup.md` — backup runbook
- `~/system/docs/runbooks/lightrag-health-monitoring.md` — health monitoring
- `~/system/docs/runbooks/ollama-cloudflare-tunnel.md` — tunnel setup

---

*Reconciliation completed 2026-06-30. No VM access performed during this documentation pass.*