LightRAG retrieval-quality recovery — MC #900176

LightRAG retrieval-quality recovery — MC #900176

Status: LIVE PATCHED / guarded rollout; agent routing and backlog remain disabled Date: 2026-08-23 Worktree: /Users/makinja/system/.claude/worktrees/lightrag-recovery-900176

Verified starting point

The Cloudflare route, authentication, health endpoint, text upload, background processing, and vector write path work. The failure is higher in the stack:

1. The MC #900165 source is present as a processed full document, two text chunks, and two vector entries.
2. For the natural query “What does SCR mean in MC 900165 IndyDevDan smart prompting plan?”, the correct chunk ranks third at cosine 0.5319866538.
3. Retrieval-only output contains the correct definition SCR = simplify/compress/repeat.
4. Server-side synthesis nevertheless answers Simplify, Clarify, Refine and fabricates /path/to/indidevdan_smart_prompting_plan_section_scr.md.
5. A generated-answer request poisons a later retrieval-only request for the same query/mode: LightRAG core 1.3.4 computes its response-cache key from only (mode, query), checks that cache before only_need_context, and can therefore return the cached hallucinated answer instead of raw context.
6. Deployed LightRAG core 1.3.4 accepts only text in POST /documents/text; client-supplied metadata is silently ignored, so current text uploads become unknown_source.
7. The old ingest queue contains 14,006 stale rows. Its adapters and drain worker remain disabled and must not be enabled or drained as part of this recovery.

Root-cause evidence: /Users/makinja/system/evidence/lightrag-test-2026-08-23/root-cause-after-client-fix.json.

Decision

Repair retrieval and provenance without trusting LightRAG’s answer generator and without mutating the existing corpus destructively. Build and validate against an isolated blue/green canary corpus first. Agent routing remains disabled until the quality gate passes.

Build scope

W1 — Provenance contract

• Add a deterministic provenance envelope inside ingested text because the deployed API ignores external metadata.
• Apply it consistently to manual CLI upload, queue drain upload, and bulk upload.
• Preserve source type, source ID/path/URL, content hash, and ingestion timestamp without including credentials or private environment data.
• Ensure already-enveloped content is not double-wrapped.

W2 — Agent-safe retrieval

• Add a retrieval-only path that never invokes server-side answer synthesis.
• Isolate retrieval-only requests from generated-answer cache entries; validate the server cache correction in the blue/green canary before any live-stack change.
• Parse returned chunks, preserve source attribution, apply deterministic lexical/exact-ID reranking, and return bounded advisory context.
• Reject or clearly mark unknown_source; never fabricate citations.
• Keep ordinary LightRAG failures fail-open for agents, but expose diagnostics.

W3 — Evaluation

• Add deterministic unit tests for provenance, parsing, reranking, source filtering, bounded output, and no-generated-answer behavior.
• Add an exact-source live canary including MC #900165.
• Record retrieval, provenance, latency, and hallucination-safe outcomes separately.

W4 — Blue/green canary corpus

• Define and run an isolated LightRAG canary with separate ports and volumes; do not touch live volumes.
• Ingest a small curated authoritative corpus only: current BookStack/runbook or equivalent local sources, active rules/specs, and selected completed MC outcomes.
• Exclude session dumps, runtime evidence noise, credentials, and the 14,006-row stale queue.
• Tear down or keep stopped after evidence capture unless explicitly approved for cutover.

Non-goals

• No blind backlog replay.
• No automatic per-prompt injector.
• No destructive edit of the current graph or Docker volumes.
• No embedding-model switch before measurement.
• No claim that LightRAG is safe merely because HTTP/status checks pass.

Acceptance criteria

1. All changed JavaScript parses and relevant tests pass.
2. Every new canary document has deterministic source provenance in retrieved context.
3. The MC #900165 exact query returns the correct source and SCR = simplify/compress/repeat without server synthesis or invented citations.
4. Controlled-corpus Recall@5 is at least 0.80, provenance coverage is 100%, and exact canaries have zero fabricated references.
5. Context is bounded and retrieval failures remain non-blocking for agents.
6. Live LightRAG volumes and the 14,006-row stale queue are unchanged by the canary run.
7. Pi/Claude global LightRAG routing remains disabled until independent verification passes.
8. Builder and independent validator evidence are recorded before MC #900176 can move to review.

Rollback

Code rollback is one commit revert. Canary rollback is docker compose down against the dedicated canary compose project; its separate named volumes can be preserved for evidence or deleted only with explicit approval. For the live server cache patch, rollback requires reverting the patch commits and force-recreating only the lightrag container from the image; a plain restart retains the already-patched container filesystem. Live data volumes must never be removed.

AFTER — implementation and guarded live deployment

Shipped

• Deterministic, allowlisted in-text provenance: lib/lightrag-provenance.js.
• Retrieval-only parsing, exact ID/acronym reranking, token-bound matching, definition-centered excerpts, unknown-source handling, and bounded context: lib/lightrag-safe-context.js.
• Provenance wired into manual upload, queue drain, and bulk upload clients.
• safe-query never invokes server synthesis and rejects generated-answer-shaped cache collisions.
• discover.js now uses raw naive context, retains enough chunks for attribution, reranks locally, and centers its summary on the answer.
• Runtime LightRAG operate.py patch bypasses generated-answer cache reads for exactly four context/prompt query paths; it is idempotent and fails closed on upstream drift.
• Isolated canary compose uses loopback 127.0.0.1:19621 and run-scoped lr900176-canary-* volumes.

Live commits: c55a98599, a92d8868c, 5010a8ccc, b6d3ad7d6, 79778b950, 2dbebc585, 82eb4641d, 61d23a9df.

Verification

• Unit tests: 13/13 Node PASS plus 3/3 Python PASS.
• Isolated five-document canary: Recall@5 1.00, provenance coverage 1.00, exact-term pass 1.00, synthesis calls 0, p95 1064ms.
• Cache-poisoning probe: a generated query completed first; the same retrieval-only query still returned raw attributed context — PASS.
• Live lightrag and lightrag-neo4j: running and healthy; runtime cache-isolation markers: 4.
• Controlled live upload increased processed documents 28,748 → 28,749.
• Exact live query: authoritative MC #900165 report ranked first, 100% selected-source provenance, SCR = simplify/compress/repeat, server synthesis disabled.
• Live discover.js: shows that definition with [type=manual] [company=alai] [file=~/system/reports/indydevdan-agent-improvements-2026-08-23.md].
• Queue remains exactly 14,006 rows / 37,466,092 content bytes; all five ingest/drain LaunchAgents remain NOT_LOADED.
• Pi and Claude Code now have on-demand safe-query routing in their global prompts and lightrag-query skills. Automatic per-prompt injection remains absent. Fresh Pi and fresh Claude Code sessions both resolved MC #900165 via the exact safe command, attributed the report, returned SCR = simplify/compress/repeat, and did not invoke generated query modes or --allow-unknown.
• Independent implementation review: PASS, P0=0, P1=0. Independent post-deploy review: core/live verdict PASS; its bookkeeping/routing questions are resolved by resuming MC #900176 and this explicit guarded-rollout decision.

Primary evidence:

• /Users/makinja/system/evidence/900176/canary-pass1/verdict.json
• /Users/makinja/system/evidence/900176/canary-pass1/cache-isolation-probe.json
• /Users/makinja/system/evidence/900176/live-deploy/live-safe-query-attributed-final.json
• /Users/makinja/system/evidence/900176/live-deploy/live-discover-lightrag-final.txt
• /Users/makinja/system/evidence/900176/live-deploy/postdeploy-queue.json
• /Users/makinja/system/evidence/900176/live-deploy/postdeploy-ingest-services.txt
• /Users/makinja/system/evidence/900176/pi-safe-routing-canary-v4-summary.json
• /Users/makinja/system/evidence/900176/cc-safe-routing-canary-v2-summary.json
• /Users/makinja/system/evidence/900176/claude-independent-review.txt
• /Users/makinja/system/evidence/900176/claude-independent-postdeploy-review-final.txt
• /Users/makinja/system/evidence/900176/native-peer-routing-verify.txt
• /Users/makinja/system/evidence/900176/live-safe-query-bounded-final.json
• /Users/makinja/system/evidence/900176/native-peer-bounded-output-verify.txt

Deliberately not enabled

• No automatic per-prompt Pi/CC injection.
• No drain or replay of the 14,006-row stale queue.
• No activation of BookStack/MC/filesystem ingest daemons.
• The legacy lightrag-ingest-mc-directory.js path is not loaded; it remains a separate cleanup candidate before any broad ingest reactivation.

The safe on-demand command is:

node ~/system/tools/lightrag.js safe-query "question" --json

Only this retrieval-only path is approved for advisory use. Generated lightrag.js query answers remain untrusted.