Skip to main content

LightRAG G1 complete blue-green rebuild — MC #900195

LightRAG G1 complete blue/green rebuild — MC #900195

Status: IN PROGRESS / remediation gate blocked; no full ingest or cutover Date: 2026-08-24 Worktree: /Users/makinja/system/.claude/worktrees/lightrag-g1-900195

Decision

Do not repair or drain the existing G0 corpus in place. Build a separate G1 generation, validate it, then cut over with G0 preserved as rollback. Pi and Claude Code continue using retrieval-only safe-query; server-generated answers and automatic per-prompt injection remain disabled.

Verified baseline

  • Live G0: LightRAG core 1.3.4, healthy, 28,749 processed statuses, 28,751 full-doc records, 21,267 text chunks, 85,807 entities.
  • Attribution: only 546/28,751 full documents (1.899%) and 387/21,267 chunks (1.82%) are attributed.
  • Frozen queue: 14,006 unique hashes / 37,466,092 content bytes, all queued, dated 2026-04-16 through 2026-07-31.
  • The frozen queue contains 1,193 superseded source versions; among 4,500 latest local evidence/spec/rule identities, 343 files are gone and 31 hashes are stale.
  • All six broad ingest/integrity LaunchAgents are NOT_LOADED.
  • Safe on-demand retrieval is live from MC #900176, but the historical corpus remains too unattributed for complete coverage.

Inventory: /Users/makinja/system/evidence/900176/full-recovery-inventory-2026-08-24.json.

Upgrade target

Use the official stable HKUDS release, not latest and not the v1.5.7rc2 release candidate:

  • Version: v1.5.6
  • Image: ghcr.io/hkuds/lightrag:v1.5.6
  • Multi-arch digest: sha256:ab23a9c83a735901b18c8960b6b482b602d5b6291abb7e07c5776f7bb2da504e
  • ARM64 manifest: sha256:65f210a9f5fb41d64c24ca4979aa329f617b1fac26105c370853bfcf2cb046bf

The source contract was verified directly at tag v1.5.6: text insertion requires valid file_source, context-only returns before answer-cache handling, query results include structured raw data/references, and custom chunking is supported.

Build waves

G1.1 — Official isolated runtime

  • Dedicated lr-g1-900195-* volumes/network and loopback port.
  • Official image pinned by digest; no G0 runtime monkey patches.
  • Existing local Ollama and BGE-M3; Neo4j isolated from G0.
  • Prove health, auth mode, native file_source, structured context, cache separation, restart persistence, and zero G0 mutation.

G1.2 — Canonical source registry

Versioned policy with include/exclude rules and stable source IDs:

  • Include current BookStack pages, active rules/specs, completed MC outcomes, and explicitly curated durable evidence.
  • Exclude sessions, runtime/generated evidence, logs, backups, archives, secrets, vendor/build output, and stale/superseded versions.
  • Each source gets native file_source, in-text provenance envelope, content hash, source timestamp, and authority class.

G1.3 — Regenerated latest-only queue

  • Build ingest-queue-g1.sqlite from current authoritative sources.
  • One latest row per canonical source identity/chunk.
  • Never copy or drain G0's 14,006 queue.
  • Deterministic manifest and source counts; queue remains reproducible.

G1.4 — Model and throughput benchmark

  • Small representative corpus first.
  • Compare available local extraction models only if necessary; keep BGE-M3 fixed.
  • Select the fastest model that passes extraction and retrieval gates.
  • Serial/backpressured ingestion with restart/resume evidence.

G1.5 — Full curated ingest and evaluation

  • 35–50 gold queries across rules, infrastructure, products, MC outcomes, and specs.
  • Negative tests for stale GCP Bilko facts, unknown sources, session noise, auth errors, and cache poisoning.
  • Retrieval-only output remains the agent contract.

G1.6 — Cutover and monitoring

  • Switch local/Cloudflare origin only after independent PASS.
  • Verify health plus authenticated real safe-query, source attribution, latency, freshness, and no queue growth.
  • Preserve G0 stopped with all volumes for immediate rollback.

Current progress — 2026-08-24

  • BookStack-first takeover verified at 13:55 CEST. MC #900195 lease moved from session 01a025b8-d67a-7c05-a87e-bdd52d98ad9b to session 01a03398-47e3-7a20-8186-71d1f3c0205b; current files and machine evidence, not the prior transcript, are authoritative.
  • G1.1 official runtime remains live in isolation on 127.0.0.1:19631: core 1.5.6, API 0328, pinned official image, separate lr-g1-900195-* volumes, restart persistence PASS, G0 unchanged.
  • Native file_source is required and verified; /query/data returns structured chunks/references and remains retrieval-only after a generated-answer cache entry.
  • Five-document official-image canary: Recall@5 1.00, provenance 1.00, exact terms 1.00, synthesis calls 0, p95 200ms.
  • Canonical queue v4 exists at /Users/makinja/system/state/lightrag-g1/ingest-queue-g1-900195-v4.sqlite: 1,617 pending alai-shared documents, unique source/native/hash IDs, zero unknown sources and zero reported deep-secret/PII findings. It is not cleared for ingest: a direct read-only query still finds 112 /archive/ or /_archive/ rows, and the product freshness rule uses an unsupported fixed date cutoff.
  • Official LightRAG stays unpatched. A pinned-image sidecar injects top-level think:false into Ollama /api/chat; direct control returned thinking_chars=0, and proxy counters proved requests were rewritten without logging bodies.
  • The fresh qwen3:8b-q8_0 benchmark ended, not passed: exit 143 after 3,222 seconds; queue state is 11 processed, 1 failed, 1 accepted, 7 pending. The run exposed observer-timeout misclassification, oversized unchunked inputs, and extraction truncation at OLLAMA_LLM_NUM_PREDICT=1200.
  • Full ingest, 35–50 query evaluation, edge exposure, rollback demonstration, and cutover have not started. Remediation must first close independent findings R2-F1 through R2-F8 and produce a fresh 20/20 benchmark with zero failures, timeouts, or truncated-extraction warnings.
  • Code commits currently in the task worktree: 11c7782c0, 7895c760c, 3c8a49607, 35ccd7d41.

Acceptance

  1. Official pinned G1 runtime starts without G0 patches and survives restart.
  2. G0 container IDs, volumes, corpus counts, and 14,006-row queue remain unchanged during G1 build.
  3. New corpus has 100% native file-source/provenance coverage and zero unknown_source in top-5.
  4. Latest-only queue contains no missing files, stale hashes, or superseded source versions.
  5. Recall@5 ≥ 0.85, MRR ≥ 0.65, exact-term coverage ≥ 0.90, stale-contradiction rate 0, p95 retrieval ≤ 3s.
  6. BookStack/MC freshness canaries complete within 15 minutes after steady-state adapters are enabled.
  7. Fresh Pi and Claude Code use only attributed safe-query; generated-answer modes and --allow-unknown are not used for answers.
  8. Independent implementation and post-cutover verification pass with P0=0/P1=0.
  9. G0 rollback is demonstrated before G1 becomes the sole live generation.

Hard safety boundaries

  • Never run the existing 14,006-row drain.
  • Never attach G1 to lr106747-* volumes.
  • Never delete G0 volumes during this task.
  • Never expose credentials in compose, evidence, logs, or prompts.
  • No production cutover before the full evaluation and independent verifier pass.

Rollback

Before cutover, stop and remove only G1 containers/network; retain its volumes for evidence. After cutover, revert Cloudflare/local origin to G0 and start the unchanged G0 compose. Volume deletion is out of scope and requires separate explicit approval.

Back to top