Skip to main content

LightRAG G1 complete blue-green rebuild — MC #900195

LightRAG G1 complete blue/green rebuild — MC #900195

Status: IN PROGRESS / remediationclean gateG1 blocked;benchmark running; no full ingest or cutover Date: 2026-08-24 Worktree: /Users/makinja/system/.claude/worktrees/lightrag-g1-900195

Decision

Do not repair or drain the existing G0 corpus in place. Build a separate G1 generation, validate it, then cut over with G0 preserved as rollback. Pi and Claude Code continue using retrieval-only safe-query; server-generated answers and automatic per-prompt injection remain disabled.

Verified baseline

  • Live G0: LightRAG core 1.3.4, healthy, 28,749 processed statuses, 28,751 full-doc records, 21,267 text chunks, 85,807 entities.
  • Attribution: only 546/28,751 full documents (1.899%) and 387/21,267 chunks (1.82%) are attributed.
  • Frozen queue: 14,006 unique hashes / 37,466,092 content bytes, all queued, dated 2026-04-16 through 2026-07-31.
  • The frozen queue contains 1,193 superseded source versions; among 4,500 latest local evidence/spec/rule identities, 343 files are gone and 31 hashes are stale.
  • All six broad ingest/integrity LaunchAgents are NOT_LOADED.
  • Safe on-demand retrieval is live from MC #900176, but the historical corpus remains too unattributed for complete coverage.

Inventory: /Users/makinja/system/evidence/900176/full-recovery-inventory-2026-08-24.json.

Upgrade target

Use the official stable HKUDS release, not latest and not the v1.5.7rc2 release candidate:

  • Version: v1.5.6
  • Image: ghcr.io/hkuds/lightrag:v1.5.6
  • Multi-arch digest: sha256:ab23a9c83a735901b18c8960b6b482b602d5b6291abb7e07c5776f7bb2da504e
  • ARM64 manifest: sha256:65f210a9f5fb41d64c24ca4979aa329f617b1fac26105c370853bfcf2cb046bf

The source contract was verified directly at tag v1.5.6: text insertion requires valid file_source, context-only returns before answer-cache handling, query results include structured raw data/references, and custom chunking is supported.

Build waves

G1.1 — Official isolated runtime

  • Dedicated lr-g1-900195-* volumes/network and loopback port.
  • Official image pinned by digest; no G0 runtime monkey patches.
  • Existing local Ollama and BGE-M3; Neo4j isolated from G0.
  • Prove health, auth mode, native file_source, structured context, cache separation, restart persistence, and zero G0 mutation.

G1.2 — Canonical source registry

Versioned policy with include/exclude rules and stable source IDs:

  • Include current BookStack pages, active rules/specs, completed MC outcomes, and explicitly curated durable evidence.
  • Exclude sessions, runtime/generated evidence, logs, backups, archives, secrets, vendor/build output, and stale/superseded versions.
  • Each source gets native file_source, in-text provenance envelope, content hash, source timestamp, and authority class.

G1.3 — Regenerated latest-only queue

  • Build ingest-queue-g1.sqlite from current authoritative sources.
  • One latest row per canonical source identity/chunk.
  • Never copy or drain G0's 14,006 queue.
  • Deterministic manifest and source counts; queue remains reproducible.

G1.4 — Model and throughput benchmark

  • Small representative corpus first.
  • Compare available local extraction models only if necessary; keep BGE-M3 fixed.
  • Select the fastest model that passes extraction and retrieval gates.
  • Serial/backpressured ingestion with restart/resume evidence.

G1.5 — Full curated ingest and evaluation

  • 35–50 gold queries across rules, infrastructure, products, MC outcomes, and specs.
  • Negative tests for stale GCP Bilko facts, unknown sources, session noise, auth errors, and cache poisoning.
  • Retrieval-only output remains the agent contract.

G1.6 — Cutover and monitoring

  • Switch local/Cloudflare origin only after independent PASS.
  • Verify health plus authenticated real safe-query, source attribution, latency, freshness, and no queue growth.
  • Preserve G0 stopped with all volumes for immediate rollback.

Current progress — 2026-08-24

  • BookStack-first takeover verified at 13:55 CEST. MC #900195 lease moved from session 01a025b8-d67a-7c05-a87e-bdd52d98ad9b to session 01a03398-47e3-7a20-8186-71d1f3c0205b; current files and machine evidence, not the prior transcript, are authoritative.
  • G1.1Fusion officialHarness completed two bounded builder/validator loops. Commits 502b512c4 and d0d6e272f close archive leakage, observer-timeout misclassification, fixed-date freshness aging, dotted Norwegian bank-account scanning, proxy readiness, Python bytecode cleanup, bounded prechunking, stale queue defaults, and automatic truncation-log preflight. Both acceptance gates passed.
  • Canonical v6 queue at /Users/makinja/system/state/lightrag-g1/ingest-queue-g1-900195-v6.sqlite has 4,405 pending rows. Independent rerun: verifier PASS; unique source/native/hash IDs 4,405/4,405/4,405; archive 0; oversized 0; unknown source 0; sensitive 0; disallowed BookStack books 0; maximum final content length 13,875 under the committed 16,000 bound. Relevant tests: Node 49/49, Python 7/7.
  • G1 isolated runtime remainswas liveforce-recreated inwithout isolationdeleting onvolumes: 127.0.0.1:19631:official core 1.5.6, API 0328, pinnedsidecar officialupstream image,readiness separateready, qwen3:8b-q8_0, think:false, output cap 2,400, overall LLM caps 4,096. G1 Neo4j and all G0 container IDs stayed unchanged; G0 remains healthy at 28,749 processed documents.
  • Frozen G0 queue remains exactly 14,006 rows / 37,466,092 content bytes, all queued. No G0 queue, container, or volume was drained, recreated, or deleted.
  • Fresh benchmark v2 started at 15:22 CEST in tmux lr-g1-900195-*bench-v2-900195 volumes,from restartthe persistenceclean PASS,v6 G0queue. unchanged.
    • At
  • Nativethe file_source15:32 ischeckpoint: required3 andprocessed, verified;1 /query/dataprocessing/accepted, returns16 structuredpending; chunks/referencesproxy and remains retrieval-only after a generated-answer cache entry.
  • Five-document official-image canary: Recall@5 1.00, provenance 1.00, exact terms 1.00, synthesis callserrors 0, p95and every observed chat request had 200msthink:false.
  • CanonicalEvidence queue v4 exists atdirectory: /Users/makinja/system/state/lightrag-g1/ingest-queue-g1-900195-v4.sqlite: 1,617 pending alai-shared documents, unique source/native/hash IDs, zero unknown sources and zero reported deep-secret/PII findings. It is not cleared for ingest: a direct read-only query still finds 112 /archive/ or /_archive/ rows, and the product freshness rule uses an unsupported fixed date cutoff.
  • Official LightRAG stays unpatched. A pinned-image sidecar injects top-level think:false into Ollama /api/chat; direct control returned thinking_chars=0, and proxy counters proved requests were rewritten without logging bodies.
  • The fresh qwen3:8b-q8_0 benchmark ended, not passed: exit 143 after 3,222 seconds; queue state is 11 processed, 1 failed, 1 accepted, 7 pending. The run exposed observer-timeout misclassification, oversized unchunked inputs, and extraction truncation at OLLAMA_LLM_NUM_PREDICT=1200evidence/900195/benchmark-v2/.
  • R2-F1 remains open until that exact fresh run finishes with 20 processed, 0 failed, 0 observation timeouts, and truncation preflight warnings 0. Full ingest, 35–50 query evaluation, edge exposure, rollback demonstration, and cutover have not started. Remediation must first close independent findings R2-F1 through R2-F8 and produce a fresh 20/20 benchmark with zero failures, timeouts, or truncated-extraction warnings.
  • Code commits currently in the task worktree: 11c7782c0, 7895c760c, 3c8a49607, 35ccd7d41.

Acceptance

  1. Official pinned G1 runtime starts without G0 patches and survives restart.
  2. G0 container IDs, volumes, corpus counts, and 14,006-row queue remain unchanged during G1 build.
  3. New corpus has 100% native file-source/provenance coverage and zero unknown_source in top-5.
  4. Latest-only queue contains no missing files, stale hashes, or superseded source versions.
  5. Recall@5 ≥ 0.85, MRR ≥ 0.65, exact-term coverage ≥ 0.90, stale-contradiction rate 0, p95 retrieval ≤ 3s.
  6. BookStack/MC freshness canaries complete within 15 minutes after steady-state adapters are enabled.
  7. Fresh Pi and Claude Code use only attributed safe-query; generated-answer modes and --allow-unknown are not used for answers.
  8. Independent implementation and post-cutover verification pass with P0=0/P1=0.
  9. G0 rollback is demonstrated before G1 becomes the sole live generation.

Hard safety boundaries

  • Never run the existing 14,006-row drain.
  • Never attach G1 to lr106747-* volumes.
  • Never delete G0 volumes during this task.
  • Never expose credentials in compose, evidence, logs, or prompts.
  • No production cutover before the full evaluation and independent verifier pass.

Rollback

Before cutover, stop and remove only G1 containers/network; retain its volumes for evidence. After cutover, revert Cloudflare/local origin to G0 and start the unchanged G0 compose. Volume deletion is out of scope and requires separate explicit approval.

Back to top