Skip to main content

RAG Queue Recovery — MC #106978

RAG Queue Recovery — MC #106978

Status

S1–S4 are complete. S5 is technically ready for a fresh, time-bounded G4 decision but remains offline and unauthorized for production effects. S6 requires a 19-day recurrence soak; S7 closes the postmortem only after Proveo PASS.

Data-accounting result

  • 47,322 source records are accounted for.
  • 14,003 were recovered, 3 were replayable, 22,336 were semantic duplicates, and 10,980 malformed records remain byte-preserved.
  • All 82 uncertain records have evidence-backed dispositions.
  • Unexplained loss: 0. Irretrievable loss: 0.
  • The untrusted historical generation is custody evidence only and is never a service or rollback source.

Frozen S5 inputs

  • Final client: 33d6e30545bdc6de171732a069316c6a05ec5f9e.
  • Final server endpoint: 508e618b6c071564e07bcea3530c6826c5c142b9.
  • Preactivated candidate: f26fae393e46e40d01721f9423fc8036cea250a89570d4fb5619638a78d61740.
  • Safe rollback: f50eeb7a3d8d575a016f3f00eb906417aa275f529842d32d1b424045b07185b6.
  • Candidate accounting: 112 receipt-backed accepted rows and 13,894 untouched queued rows.
  • Receipt set: 112 private schema-v3 records bound to the actual public runtime endpoint.

Proveo, an explicit P2P peer, and a separate Securion witness all passed with no P0/P1/P2 findings. These are offline readiness verdicts, not deployment authority.

G4 sequence

  1. Record exact CEO authority and a maximum 45-minute UTC control window.
  2. Deploy the exact server image first; prove authentication, one bounded document-ID lookup, no enumeration, and deterministic rollback.
  3. Atomically deploy the three client runtime files, the private schema-v3 receipts, and the preactivated candidate.
  4. Reconcile the 112 accepted rows using authenticated bounded GET only. This phase hard-fences every POST path, including the optional backlog nudge.
  5. Start the watchdog, then run one downstream canary each for Mission Control, filesystem, and BookStack.
  6. Start the persistent drain at no more than 20 POSTs/minute, then restore the three source adapters serially. The superseded legacy outbox-ingest job remains disabled.
  7. Replay the seven authoritative MC outcomes plus a mandatory final structured delta sweep.

Terminal semantics

  • Only exact PROCESSED may mark a row done.
  • Exact FAILED is terminal.
  • Absent, malformed, oversized, unavailable, PENDING, and PROCESSING remain accepted and are never posted again.
  • MD5 is used only to derive the deployed server lookup key; SHA-256 content and delivery identities remain semantic authority.

Rollback

Any wrong hash/image/router, unauthorized method, accepted-only POST, queued-row mutation, activation failure, writer overlap, SQLite failure, missing audit acknowledgement, canary failure/duplicate, privacy leak, unexplained replay delta, or window expiry triggers rollback. Stop all restored jobs, preserve the failed main/WAL/SHM coherently, restore exact f50e..., restore prior client bytes and the pinned prior server image, then prove the offline/job/health fences. Never use the untrusted historical generation.

Monitoring and closure

After successful S5, Proveo owns S6 verification for the required 19-day recurrence interval. The soak repeatedly checks immutable snapshots, queue progress, one-writer topology, daemon health, corruption signatures, and downstream canaries. Main task #106978 remains open until S6 and S7 pass without force.

Back to top