Skip to main content

RAG Queue Recovery — MC #106978

RAG Queue Recovery — MC #106978

Status

S1–S4 are complete. The S5 isoffline technicallypackage readywas rebuilt by MC #107082 for athe fresh,reviewed time-boundedd38f... G4client decisionsuccessor butand exact 508e... server. It remains offlineoffline, has no authority receipt or UTC window, and is unauthorized for production effects. S6 requires a 19-day recurrence soak;soak after a separately authorized successful G4; S7 closes the postmortem only after Proveo PASS.

Data-accounting result

  • 47,322 source records are accounted for.
  • 14,003 were recovered, 3 were replayable, 22,336 were semantic duplicates, and 10,980 malformed records remain byte-preserved.
  • All 82 uncertain records have evidence-backed dispositions.
  • Unexplained loss: 0. Irretrievable loss: 0.
  • The untrusted historical generation is custody evidence only and is never a service or rollback source.

Frozen S5 inputs

  • Final client: d38f399067113dc1b29e38a315442362e3f89ee2, direct successor of 33d6e30545bdc6de171732a069316c6a05ec5f9e (patch 9b94181cce3c30f8be3946f7f0eb91bee5768deb0a51ff5a08450071c960f7fe).
  • Final server endpoint: 508e618b6c071564e07bcea3530c6826c5c142b9.
  • Preactivated candidate: f26fae393e46e40d01721f9423fc8036cea250a89570d4fb5619638a78d61740.
  • Safe rollback: f50eeb7a3d8d575a016f3f00eb906417aa275f529842d32d1b424045b07185b6.
  • Candidate accounting: 112 receipt-backed accepted rows and 13,894 untouched queued rows.
  • Receipt set: 112 private schema-v3 records bound to the actual public runtime endpoint.

Proveo,Independent anround-13 explicit P2P peer, andQA, a separate Securionreliability witnesswitness, native task-bound P2P, and clean post-commit verification all passed withfor nod38f... P0/P1/P2(294/294 findings.tests under both umask 0022 and 0077). These are offline readiness verdicts, not deployment authority. The actual qualified topology remains local Docker through the Cloudflare tunnel; the Azure VM is deallocated and the historical Azure body below its supersession notice remains documentation drift.

G4 sequence

  1. Record exact CEO authority and a maximum 45-minute UTC control window.
  2. Deploy the exact server image first; prove authentication, one bounded document-ID lookup, no enumeration, and deterministic rollback.
  3. AtomicallyCoherently deploy all staged runtime/helper bytes: the threefive clientchanged runtimepaths files,(agents/hivemind/hivemind.js, lib/rag-db-writer-lock.js, lib/rag-outbox.js, tools/lightrag-auth-helper.js, and tools/rag-drain-worker.js) and their unchanged staged receipt/spool/schema/config/HiveMind dependencies. Publish the private schema-v3 receipts,receipts and preactivated candidate with descriptor, file, and parent-directory fsync.
  4. Only after the preactivatedfuture candidate.authority is consumed, quarantine legacy system/config/.lightrag-token-cache.json if it is a UID-owned, mode-0600, single-link regular file. The runtime cache policy is exclusively state/runtime-auth/lightrag-token-cache.json under a mode-0700 parent; absence is valid and any present file must be UID-owned, mode 0600, single-link, descriptor-bound. Rollback restores both cache prestates byte-for-byte.
  5. Reconcile the 112 accepted rows using authenticated bounded GET only. This phase hard-fences every POST path, includingfallback, history/global-artifact access, global permit-state access, and the optional backlog nudge.
  6. Start the watchdog,watchdog thenand runrequire a hard local-audit acknowledgement plus a side-effect-free HiveMind local-gate-ack.
  7. After the future receipt/window exists, generate exactly three private selector files, one downstream canary each for Mission Control, filesystem, and BookStack. Each binds task #107020, the authority digest, exact delivery, and the maximum 45-minute window. Reserve the exact DB row before one-use selector consumption; never put selector or delivery identity in argv or public evidence.
  8. Start the persistent drain at no more than 20 POSTs/minute,POSTs per rolling 60 seconds, then restore the three source adapters serially. Login, nudge, Slack, canary, adapter, replay, and drain POSTs share the same durable permit state across process restart. Local audit and HiveMind are hard; Slack is retry-soft only under the literal policy. The superseded legacy outbox-ingest job remains disabled.
  9. Replay the seven authoritative MC outcomes plus a mandatory final structured delta sweep.

Terminal semantics

  • Only exact PROCESSED may mark a row done.
  • Exact FAILED is terminal.
  • Absent, malformed, oversized, unavailable, PENDING, and PROCESSING remain accepted and are never posted again.
  • MD5 is used only to derive the deployed server lookup key; SHA-256 content and delivery identities remain semantic authority.

Rollback

Any wrong hash/image/router, unauthorized method, accepted-only POST, queued-rowqueued/history/global mutation, activation or auth-cache-policy failure, writer overlap, SQLite failure, missing auditaudit/HiveMind acknowledgement, canary failure/reservation/consumption failure or duplicate, privacy leak, unexplained replay delta, or window expiry triggers rollback. Stop all restored jobs, preserve the failed main/WAL/SHM coherently, restore exact f50e..., restore all prior clientstaged-path bytesbytes, both cache prestates, receipts, and the pinned prior server image, then prove thedescriptor, fsync, offline/job/job, router, and health fences. Never use the untrusted historical generation.

Monitoring and closure

After a separately authorized successful S5, Proveo owns S6 verification for the required 19-day recurrence interval. The soak repeatedly checks immutable snapshots, queue progress, one-writer topology, durable permit capacity, daemon health, corruption signatures, auth-cache privacy, and downstream canaries. Main task #106978 remains open until S6 and S7 pass without force. BookStack publication remains blocked/drifted; this local file is canonical until an authenticated publication is separately authorized and verified.

Back to top