Skip to main content

LightRAG G1 complete blue-green rebuild — MC #900195

LightRAG G1 complete blue/green rebuild — MC #900195

Status: IN PROGRESS / clean G1 benchmark running; no full ingest or cutover Date: 2026-08-24 Worktree: /Users/makinja/system/.claude/worktrees/lightrag-g1-900195

Decision

Do not repair or drain the existing G0 corpus in place. Build a separate G1 generation, validate it, then cut over with G0 preserved as rollback. Pi and Claude Code continue using retrieval-only safe-query; server-generated answers and automatic per-prompt injection remain disabled.

Verified baseline

  • Live G0: LightRAG core 1.3.4, healthy, 28,749 processed statuses, 28,751 full-doc records, 21,267 text chunks, 85,807 entities.
  • Attribution: only 546/28,751 full documents (1.899%) and 387/21,267 chunks (1.82%) are attributed.
  • Frozen queue: 14,006 unique hashes / 37,466,092 content bytes, all queued, dated 2026-04-16 through 2026-07-31.
  • The frozen queue contains 1,193 superseded source versions; among 4,500 latest local evidence/spec/rule identities, 343 files are gone and 31 hashes are stale.
  • All six broad ingest/integrity LaunchAgents are NOT_LOADED.
  • Safe on-demand retrieval is live from MC #900176, but the historical corpus remains too unattributed for complete coverage.

Inventory: /Users/makinja/system/evidence/900176/full-recovery-inventory-2026-08-24.json.

Upgrade target

Use the official stable HKUDS release, not latest and not the v1.5.7rc2 release candidate:

  • Version: v1.5.6
  • Image: ghcr.io/hkuds/lightrag:v1.5.6
  • Multi-arch digest: sha256:ab23a9c83a735901b18c8960b6b482b602d5b6291abb7e07c5776f7bb2da504e
  • ARM64 manifest: sha256:65f210a9f5fb41d64c24ca4979aa329f617b1fac26105c370853bfcf2cb046bf

The source contract was verified directly at tag v1.5.6: text insertion requires valid file_source, context-only returns before answer-cache handling, query results include structured raw data/references, and custom chunking is supported.

Build waves

G1.1 — Official isolated runtime

  • Dedicated lr-g1-900195-* volumes/network and loopback port.
  • Official image pinned by digest; no G0 runtime monkey patches.
  • Existing local Ollama and BGE-M3; Neo4j isolated from G0.
  • Prove health, auth mode, native file_source, structured context, cache separation, restart persistence, and zero G0 mutation.

G1.2 — Canonical source registry

Versioned policy with include/exclude rules and stable source IDs:

  • Include current BookStack pages, active rules/specs, completed MC outcomes, and explicitly curated durable evidence.
  • Exclude sessions, runtime/generated evidence, logs, backups, archives, secrets, vendor/build output, and stale/superseded versions.
  • Each source gets native file_source, in-text provenance envelope, content hash, source timestamp, and authority class.

G1.3 — Regenerated latest-only queue

  • Build ingest-queue-g1.sqlite from current authoritative sources.
  • One latest row per canonical source identity/chunk.
  • Never copy or drain G0's 14,006 queue.
  • Deterministic manifest and source counts; queue remains reproducible.

G1.4 — Model and throughput benchmark

  • Small representative corpus first.
  • Compare available local extraction models only if necessary; keep BGE-M3 fixed.
  • Select the fastest model that passes extraction and retrieval gates.
  • Serial/backpressured ingestion with restart/resume evidence.

G1.5 — Full curated ingest and evaluation

  • 35–50 gold queries across rules, infrastructure, products, MC outcomes, and specs.
  • Negative tests for stale GCP Bilko facts, unknown sources, session noise, auth errors, and cache poisoning.
  • Retrieval-only output remains the agent contract.

G1.6 — Cutover and monitoring

  • Switch local/Cloudflare origin only after independent PASS.
  • Verify health plus authenticated real safe-query, source attribution, latency, freshness, and no queue growth.
  • Preserve G0 stopped with all volumes for immediate rollback.

Current progress — 2026-08-24

  • BookStack-first takeover verified at 13:55 CEST. MC #900195 lease movedis fromheld session 01a025b8-d67a-7c05-a87e-bdd52d98ad9b toby session 01a03398-47e3-7a20-8186-71d1f3c0205b; current files and machine evidence, not the prior transcript,transcripts, are authoritative.
  • Fusion Harness completed two bounded builder/validator loops. Commits 502b512c4 and d0d6e272f close archive leakage, observer-timeout misclassification, fixed-date freshness aging, dottedsensitive-data Norwegian bank-account scanning,gaps, proxy readiness, Python bytecode cleanup, bounded prechunking, stale queue defaults, and automatic truncation-log preflight. BothFollow-up acceptancecommits gatesed0df54ba passed.and 244146959 apply measured extraction-limit iterations.
  • Benchmark v2 completed all 20 documents with zero failed/timeouts, but preflight correctly blocked it on five done_reason=length warnings at output cap 2,400. Benchmark v3 at output cap 4,096 again completed 20/20 with zero failed/timeouts, but remained blocked on four length warnings. Neither result was promoted as PASS.
  • The v3 warnings were sanitized and correlated to three dense source documents without recording model/document bodies. Because raising the output cap did not eliminate truncation, the per-request server chunk size was reduced from 800 to 400 tokens.
  • Canonical v6v7 queue at /Users/makinja/system/state/lightrag-g1/ingest-queue-g1-900195-v6.v7.sqlite has 4,405407 pending rows.rows Independent rerun:and verifier PASS;PASS: unique source/native/hash IDs 4,405/407/4,405/407/4,405407; archive 0; oversized 0; unknown source 0; sensitive 0; disallowed BookStack books 0; maximum final content length 13,875; underevery therow committeduses server chunk size 16,000400.
  • Fresh benchmark v4 started at 20:49 CEST in tmux lr-g1-bench-v4-900195 bound.with Relevanta tests:unique Nodesource label. Initial checkpoint: 1 processing/accepted, 19 pending. Evidence directory: 49/49, Python 7/7/Users/makinja/system/evidence/900195/benchmark-v4/.
  • G1 remains isolated runtimeand washealthy force-recreated without deleting volumes:on official core 1.5.6, API 0328, sidecar upstream readiness ready, qwen3:8b-q8_0,; think:false, output cap 2,4004,096, overall LLM caps 4,096. G1 Neo4j and allsidecar G0upstream containerreadiness IDsremain stayedactive. unchanged; G0 remains healthy at 28,749 processed documents.
  • Frozen G0 queue remains exactly 14,006 rows / 37,466,092 content bytes, all queued.queued; Nono G0 queue, container, or volume was drained, recreated,drained or deleted.
  • Fresh benchmark v2 started at 15:22 CEST in tmux lr-g1-bench-v2-900195 from the clean v6 queue. At the 15:32 checkpoint: 3 processed, 1 processing/accepted, 16 pending; proxy errors 0, and every observed chat request had think:false. Evidence directory: /Users/makinja/system/evidence/900195/benchmark-v2/.
  • R2-F1 remains open until thatbenchmark exact fresh runv4 finishes with 20 processed, 0 failed, 0 observation timeouts, and truncation preflight warnings 0. Full ingest, 35–50 query evaluation, edge exposure, rollback demonstration, and cutover have not started.

Acceptance

  1. Official pinned G1 runtime starts without G0 patches and survives restart.
  2. G0 container IDs, volumes, corpus counts, and 14,006-row queue remain unchanged during G1 build.
  3. New corpus has 100% native file-source/provenance coverage and zero unknown_source in top-5.
  4. Latest-only queue contains no missing files, stale hashes, or superseded source versions.
  5. Recall@5 ≥ 0.85, MRR ≥ 0.65, exact-term coverage ≥ 0.90, stale-contradiction rate 0, p95 retrieval ≤ 3s.
  6. BookStack/MC freshness canaries complete within 15 minutes after steady-state adapters are enabled.
  7. Fresh Pi and Claude Code use only attributed safe-query; generated-answer modes and --allow-unknown are not used for answers.
  8. Independent implementation and post-cutover verification pass with P0=0/P1=0.
  9. G0 rollback is demonstrated before G1 becomes the sole live generation.

Hard safety boundaries

  • Never run the existing 14,006-row drain.
  • Never attach G1 to lr106747-* volumes.
  • Never delete G0 volumes during this task.
  • Never expose credentials in compose, evidence, logs, or prompts.
  • No production cutover before the full evaluation and independent verifier pass.

Rollback

Before cutover, stop and remove only G1 containers/network; retain its volumes for evidence. After cutover, revert Cloudflare/local origin to G0 and start the unchanged G0 compose. Volume deletion is out of scope and requires separate explicit approval.

Back to top