Skip to main content

LightRAG Recovery — Session 6c3cc7f0 Post-Mortem (2026-06-29)

LightRAG Recovery — Session 6c3cc7f0 Reconciliation

DocType: Incident Post-Mortem + Reconciliation Session: 6c3cc7f0-ff37-46c3-99ab-952b75a451b8 (2026-06-29) Documented by: FlowForge reconciliation pass (2026-06-30) MC Task: #104528 LightRAG drain + FORGE migration eval Incident Memo: incident_lightrag_403_extraction_and_chunkvdb_corruption_2026-06-29.md

1. Context: The "Do Not Touch" Rule

MEMORY.md contains the standing rule:

"LightRAG FIXED not dead — never re-diagnose/touch neo4j (project_lightrag_fixed_2026-06-12)."

Session 6c3cc7f0 did run approximately 26 az vm run-command invoke calls against vm-alai-lightrag (RG-ALAI-LIGHTRAG). See Flag A below for the alignment assessment.

2. Initial State (verified from transcript)

Parameter Value
VM vm-alai-lightrag, RG-ALAI-LIGHTRAG, public IP 20.240.61.67
Compose dir /home/alai-admin/lightrag/
LightRAG version api_version: 0154, core: 1.3.4
LLM_BINDING_HOST (pre-session) https://ollama.basicconsulting.no
EMBEDDING_BINDING_HOST (pre-session) http://ollama:11434 (correct, already internal)
LLM_MODEL llama3.1:8b
Internal ollama models llama3.2:1b, bge-m3:latest — llama3.1:8b was NOT present
processed 19,236
pending 40,559
failed 8–10
pipeline_busy false (stuck; pipeline not retrying)

3. Root Cause Chain (Three Links)

The session established the following confirmed chain of failures, documented in the incident memo:

.env line 10 pointed to https://ollama.basicconsulting.no, a leftover from before the domain migration to alai.no (pre-2026-05-17). The compose default in docker-compose.yml line 39 was ${LLM_BINDING_HOST:-https://ollama.basicconsulting.no}.

ollama.alai.no is served by cloudflared tunnel "mattermost" (id 3315a609-7934-45c5-ad0c-56d86d16374d). Its origin rule was pointing to http://10.0.0.2:11434 (FORGE Ollama). FORGE switched from Ollama to MLX (port 11435) on 2026-06-25. The Ollama :11434 service on FORGE was dead. The tunnel needed repointing to http://localhost:11434 (ANVIL local Ollama, which does have llama3.1:8b + bge-m3).

Link 3 — Ollama DNS-Rebind Guard (the actual 403 killer)

Ollama returns HTTP 403 to any request whose Host header is not localhost or 127.0.0.1. cloudflared by default forwards the original Host: ollama.alai.no header to the origin. This means every LightRAG extraction call received a 403 from ollama regardless of CF Access config.

Proof in transcript: curl -H 'Host: ollama.alai.no' localhost:11434/api/tags = 403 vs curl -H 'Host: localhost' localhost:11434/api/tags = 200.

Lesson (durable): cf-cache-status: DYNAMIC on a 403 response means the origin returned it — Cloudflare Access was NOT the gating layer. Do not assume 403 content-length:0 server:cloudflare means Access blocked it.

4. What the Session Did — Chronological

Step 1: Diagnosis

Ran az vm run-command invoke to read container state, docker ps, LightRAG logs, .env contents, and docker exec lightrag env. Confirmed extraction LLM host and 403 pattern.

Step 2: Wrong Fix — Internal Ollama (MY MISTAKE #1)

Pulled llama3.1:8b to the internal ollama container (docker exec ollama ollama pull llama3.1:8b, 4.9 GB). Edited .env to set LLM_BINDING_HOST=http://ollama:11434. Ran docker compose up -d lightrag.

Why this was wrong: The VM is 2 vCPU / 7.8 GB RAM / no GPU. This architecture is documented in project_lightrag_topology_fix_2026-06-03 — the LLM is remote BY DESIGN because the VM cannot host an 8B model. The result was OOM-thrash: ollama logged "llm server loading model" in a loop, never ready, ~195 MB RAM remaining. Zero docs drained.

Step 3: vdb_chunks.json Corruption (MY MISTAKE #2)

Ran docker compose up -d lightrag to recreate the container with the new .env (without a graceful stop first). The default stop timeout (10 seconds) sent SIGKILL mid-flush.

Effect: vdb_chunks.json was truncated from 132 MB to 48 MB. The matrix (chunk embeddings) was lost. On next start, nano_vectordb.load_storage raised JSONDecodeError → container entered crash-loop.

The large vdbs (vdb_entities.json 516 MB, vdb_relationships.json 759 MB) survived only by write-order luck.

Lesson (repeat of feedback_no_restart_prod_for_cosmetic_2026-06-04):

This container's graceful stop requires -t 180 (measured flush time: 3 minutes 4 seconds).
Never use default-timeout restart or docker compose up -d without a prior docker compose stop -t 180.
Verify backup integrity before any container lifecycle operation on this VM.

Step 4: Recovery from Backup

Source: ~/system/backups/lightrag/20260628-040003/lightrag-data.tar.gz (weekly cron snapshot)

Content: Valid vdb_chunks.json — 132 MB, JSON valid, matrix present, 17,234 data entries, content dated Jun 7 (consistent with entity/relationship vdbs frozen at that date).

Recovery sequence:

  1. Extract vdb_chunks.json locally from tarball.
  2. Upload to Azure Blob: plockfrontstaging/lightrag-backup/restore-20260629/vdb_chunks.json.
  3. Generate SAS URL (short TTL).
  4. On VM via az vm run-command: docker compose stop -t 180 (graceful, 3 min 4 sec measured).
  5. On VM: curl <SAS_URL> into volume mount /mnt/docker-data/docker/volumes/lightrag-data/_data/vdb_chunks.json.
  6. sha256 verify: b24174… (matched local extracted file).
  7. Update .env to correct LLM host (see step 5 below).
  8. docker compose up -d.

Verified result: t=80s RestartCount=0, Health=healthy. nano-vectordb loaded: vdb_chunks.json 17234 data — clean load, no JSONDecodeError. Available memory recovered from ~195 MB to 4.1 GB.

Step 5: Correct Fix — Cloudflared Tunnel + httpHostHeader

Critical infra fact (discovered in session): The tunnel is dashboard-managed (remote config). Editing /home/alai-admin/cloudflared/config.yml locally does nothing — cloudflared pulls remote config and logs "Updated to new configuration version=N". cloudflared tunnel ingress rule <url> reads the LOCAL file and misleads.

Fix path: CF API:

  • Endpoint: GET/PUT /accounts/d0ac2afb6bb5b298723b85a114151a04/cfd_tunnel/3315a609-7934-45c5-ad0c-56d86d16374d/configurations
  • Auth: Bitwarden item "Cloudflare Global API Key" (X-Auth-Email + X-Auth-Key). The generic "Cloudflare api token" in BW lacks tunnel/Access scope.
  • Change: Repoint ollama.alai.no origin from http://10.0.0.2:11434 → http://localhost:11434. Add originRequest.httpHostHeader: "localhost:11434" to bypass ollama's DNS-rebind guard.
  • The tunnel serves 32 ingress rules; only the target rule was changed, then full config was PUT back.

CF Access (Red Herring): IP-bypass policy already included VM 20.240.61.67/32 in BOTH the exact ollama.alai.no app (id bdc17e6a) and the *.alai.no wildcard app (id cd7cf0f0). CF Access was NOT the gating layer.

Verified after fix: VM → ollama.alai.no/api/tags = 200. /api/chat llama3.1:8b = 200 "OK". LightRAG logs showed "Merge N/E entities+relations". Zero new 403 errors.

5. Health Counts at Session Handoff

Taken from final docker exec lightrag python3 health probe in transcript:

Metric Value
llm_binding_host https://ollama.alai.no
embedding_binding_host http://ollama:11434
llm_model llama3.1:8b
processed 19,237 (incremented from 19,236 during session)
pending ~40,563–40,564
failed 8
processing 2
pipeline_busy True
RestartCount 0
403 errors post-fix 0

Note: processed incremented by 1 (+180s observation) after the correct fix was applied, confirming the pipeline was draining.

6. MC #104528 — Blocker Detail

Task: LightRAG drain + FORGE migration eval
Status: blocked (awaiting_forge — GOTCHA doc missing, /prompt-forge 104528 required before unblock)

Stated blocker in MC description: "add Cloudflare Access Bypass policy for VM egress IP 20.240.61.67/32 to ollama.alai.no app (Zero Trust, account d0ac2afb6bb5b298723b85a114151a04)"

Reconciliation note: The MC task description reflects the diagnosis made early in the session before the DNS-rebind guard was identified as the real root cause. Per the incident memo and transcript evidence, the CF Access bypass was already in place for 20.240.61.67. The actual fix (httpHostHeader + tunnel repoint) was applied and verified working within the same session. The stated MC blocker may now be resolved. This should be re-verified before the next session touches #104528.

Open sub-tracks in #104528:

  1. Evaluate migrating LightRAG + neo4j + ollama to FORGE (Mac Studio M3 Ultra 256 GB, OrbStack) to remove cloud/tunnel fragility. Prior reason to leave Mac: Docker Desktop crashed 3x on 2026-04-18. Counter: OrbStack removes that SPOF.
  2. Build LightRAG idle-watchdog: re-trigger pipeline when pending>0 && pipeline_busy=false for >N minutes. Without this, any future CF/tunnel blip will re-stick the 40k backlog indefinitely.
  3. Proveo validation task required per ZAKON PLAN before any drain claim.
  4. Skillforge BookStack documentation — this page closes that gap.

7. Flags

Flag A — "Do Not Touch" Rule Assessment {#flag-a}

The MEMORY.md rule states: "LightRAG FIXED not dead — never re-diagnose/touch neo4j (project_lightrag_fixed_2026-06-12)."

Assessment: The session's VM access was triggered by a real new incident (40k docs stuck pending, extraction 403 for an unknown period). The "FIXED" tag in MEMORY.md referred to a prior false-negative health-check (2026-06-12), not to permanent immunity from new incidents. The neo4j volumes were not touched. The touch was operationally justified.

However: The session made two compounding mistakes (internal-ollama wrong fix + non-graceful recreate) that would not have occurred if the topology documentation (project_lightrag_topology_fix_2026-06-03) had been read before acting. The "do not touch" rule exists partly to prevent exactly this class of well-intentioned damage. Standing rule recommendation: before any az vm run-command that modifies LightRAG container state, read the topology memo and the backup runbook first.

Verdict: Rule spirit violated (touch caused damage). Rule letter partially applies — the rule's "neo4j" qualifier was respected; the broader "do not re-diagnose" guidance was not.

Flag B — CF Access Service Token Plaintext Exposure {#flag-b}

During transcript tool result output (a Read call on lightrag.js source), the following values were printed in plaintext:

"lightrag.cf_access_client_id": "4248b2c109e87e09faf3fb82a90eeafd.access"
"lightrag.cf_access_client_secret": "[REDACTED — do not reprint]"

These are CF Access service token credentials for lightrag.alai.no. They were embedded in an alai_config block in the source file and were printed verbatim in the session transcript (file 6c3cc7f0-ff37-46c3-99ab-952b75a451b8.jsonl).

Risk: The transcript file is readable on-disk at ~/.claude/projects/-Users-makinja/. Any process or agent with filesystem access can read these values.

Recommended action: Rotate the CF Access service token for lightrag.alai.no (generate a new CID/CSEC pair in Cloudflare Zero Trust → Access → Service Auth). Update alai_config / Bitwarden. Sub-agent prompts should explicitly prohibit echoing secrets in output.

8. .env Configuration State at Handoff

File: /home/alai-admin/lightrag/.env

Key Value
LLM_BINDING_HOST https://ollama.alai.no
LLM_MODEL llama3.1:8b
EMBEDDING_BINDING_HOST http://ollama:11434

Backups on VM: .env.bak-pre-internal-llm-* (the wrong internal-ollama fix), .env.bak-pre-localembed-20260603 (older).

9. Architecture Notes (Durable)

  • VM compute: 2 vCPU / 7.8 GB RAM / no GPU. Cannot host any >1B parameter LLM. Remote LLM is REQUIRED.
  • Tunnel is dashboard-managed. Do not edit local config.yml. Use CF API for tunnel config changes.
  • Graceful stop requires -t 180. Default 10s SIGKILL always corrupts in-flight vdb flushes.
  • Weekly backup cron saves to ~/system/backups/lightrag/YYYYMMDD-HHMMSS/. The Jun 28 backup was what saved this recovery. The Jun 7 content date of vdb_chunks.json confirms vdb vectors have not been updated since then (entities/relationships progressed further in memory but vdb chunk index was stale — re-embed from kv_store_text_chunks.json is a deferred open item).
  • /health doc_status_counts is the authoritative metric. lightrag.js status returns false Relations: 0 — do not use it for health assessment.
  • ~/system/docs/runbooks/azure-lightrag-migration.md — full migration history + security incident (2026-06-18)
  • ~/system/docs/runbooks/lightrag-backup.md — backup runbook
  • ~/system/docs/runbooks/lightrag-health-monitoring.md — health monitoring
  • ~/system/docs/runbooks/ollama-cloudflare-tunnel.md — tunnel setup

Reconciliation completed 2026-06-30. No VM access performed during this documentation pass.

No comments to display

Back to top