Skip to main content

LightRAG retrieval-quality recovery — MC #900176

LightRAG retrieval-quality recovery — MC #900176

Status: BEFORE / approved by CEO Date: 2026-08-23 Worktree: /Users/makinja/system/.claude/worktrees/lightrag-recovery-900176

Verified starting point

The Cloudflare route, authentication, health endpoint, text upload, background processing, and vector write path work. The failure is higher in the stack:

  1. The MC #900165 source is present as a processed full document, two text chunks, and two vector entries.
  2. For the natural query “What does SCR mean in MC 900165 IndyDevDan smart prompting plan?”, the correct chunk ranks third at cosine 0.5319866538.
  3. Retrieval-only output contains the correct definition SCR = simplify/compress/repeat.
  4. Server-side synthesis nevertheless answers Simplify, Clarify, Refine and fabricates /path/to/indidevdan_smart_prompting_plan_section_scr.md.
  5. Deployed LightRAG core 1.3.4 accepts only text in POST /documents/text; client-supplied metadata is silently ignored, so current text uploads become unknown_source.
  6. The old ingest queue contains 14,006 stale rows. Its adapters and drain worker remain disabled and must not be enabled or drained as part of this recovery.

Root-cause evidence: /Users/makinja/system/evidence/lightrag-test-2026-08-23/root-cause-after-client-fix.json.

Decision

Repair retrieval and provenance without trusting LightRAG’s answer generator and without mutating the existing corpus destructively. Build and validate against an isolated blue/green canary corpus first. Agent routing remains disabled until the quality gate passes.

Build scope

W1 — Provenance contract

  • Add a deterministic provenance envelope inside ingested text because the deployed API ignores external metadata.
  • Apply it consistently to manual CLI upload, queue drain upload, and bulk upload.
  • Preserve source type, source ID/path/URL, content hash, and ingestion timestamp without including credentials or private environment data.
  • Ensure already-enveloped content is not double-wrapped.

W2 — Agent-safe retrieval

  • Add a retrieval-only path that never invokes server-side answer synthesis.
  • Parse returned chunks, preserve source attribution, apply deterministic lexical/exact-ID reranking, and return bounded advisory context.
  • Reject or clearly mark unknown_source; never fabricate citations.
  • Keep ordinary LightRAG failures fail-open for agents, but expose diagnostics.

W3 — Evaluation

  • Add deterministic unit tests for provenance, parsing, reranking, source filtering, bounded output, and no-generated-answer behavior.
  • Add an exact-source live canary including MC #900165.
  • Record retrieval, provenance, latency, and hallucination-safe outcomes separately.

W4 — Blue/green canary corpus

  • Define and run an isolated LightRAG canary with separate ports and volumes; do not touch live volumes.
  • Ingest a small curated authoritative corpus only: current BookStack/runbook or equivalent local sources, active rules/specs, and selected completed MC outcomes.
  • Exclude session dumps, runtime evidence noise, credentials, and the 14,006-row stale queue.
  • Tear down or keep stopped after evidence capture unless explicitly approved for cutover.

Non-goals

  • No blind backlog replay.
  • No automatic per-prompt injector.
  • No destructive edit of the current graph or Docker volumes.
  • No embedding-model switch before measurement.
  • No claim that LightRAG is safe merely because HTTP/status checks pass.

Acceptance criteria

  1. All changed JavaScript parses and relevant tests pass.
  2. Every new canary document has deterministic source provenance in retrieved context.
  3. The MC #900165 exact query returns the correct source and SCR = simplify/compress/repeat without server synthesis or invented citations.
  4. Controlled-corpus Recall@5 is at least 0.80, provenance coverage is 100%, and exact canaries have zero fabricated references.
  5. Context is bounded and retrieval failures remain non-blocking for agents.
  6. Live LightRAG volumes and the 14,006-row stale queue are unchanged by the canary run.
  7. Pi/Claude global LightRAG routing remains disabled until independent verification passes.
  8. Builder and independent validator evidence are recorded before MC #900176 can move to review.

Rollback

Code rollback is one commit revert. Canary rollback is docker compose down against the dedicated canary compose project; its separate named volumes can be preserved for evidence or deleted only with explicit approval. The current live stack and queue do not participate in rollback because this build must not mutate them.

Back to top