Skip to main content

Memory Retrieval Eval Harness

Memory Retrieval Eval Harness — MC #105224

Purpose

Regression-test discover.js memory "<query>" retrieval quality with a labeled set of real incident queries mapped to known memo files.

Artifacts

  • Harness: ~/system/tools/memory-eval-harness.js
  • Test set: ~/system/specs/memory-retrieval-eval/test-set-v1.json
  • Query-log audit: ~/system/specs/memory-retrieval-eval/query-log-miss-audit-2026-07-10.md
  • FreshHistorical evidence: ~/system/evidence/memory-eval-105224-verify-20260710/
  • P0-A evidence: ~/system/evidence/memory-eval-p0a-20260721/

How to run

node ~/system/tools/memory-eval-harness.js run \
  --test-set ~/system/specs/memory-retrieval-eval/test-set-v1.json \
  --output ~/system/evidence/memory-eval-$(date +%Y%m%d-%H%M%S)

Dry-run schema validation:

node ~/system/tools/memory-eval-harness.js run \
  --test-set ~/system/specs/memory-retrieval-eval/test-set-v1.json \
  --dry-run

Metrics

  • Recall@3:Recall@3/5/10/20: correct memo appears at rank <= k in the top20 3candidates files returnedrequested by discover.jsthe memory.eval harness.
  • MRR:MRR@20: reciprocal rank of the correct memo; 0 when not returned.returned in the top 20.

Important limitation:Normal discover.js memory "<topic>" hard-capscalls outputremain capped at top 3,3 sofor thisbackward compatibility. The eval harness cannotexplicitly distinguishuses rank--top 4+20, fromwhich achanges completeonly miss.truncation depth, not scoring or ordering.

Baseline run — 2026-07-10 (historical, before P0-A)

Fresh verification output: ~/system/evidence/memory-eval-105224-verify-20260710/

  • Labeled pairs: 40
  • Recall@3: 0.600 (24/40)
  • MRR: 0.546
  • Errors: 0/40
  • p50 latency: 174ms
  • p95 latency: 243ms

Query-logP0-A auditverification — 2026-07-21

NoEvidence: persistent~/system/evidence/memory-eval-p0a-20260721/

  • Labeled pairs: 40
  • Recall@3: 0.750 (30/40)
  • Recall@5: 0.825 (33/40)
  • Recall@10: 0.875 (35/40)
  • Recall@20: 0.925 (37/40)
  • MRR@20: 0.695
  • Errors: 0/40
  • p50 latency: 125ms
  • p95 latency: 138ms

The apparent Recall@3 change versus the historical baseline reflects corrected handling of documented fail-closed exit code 1 and evaluation over one top-20 candidate list. The --top path changes only truncation, not scoring or ordering; again was also classified as a generic stopword so non-topical repeated-query wording remains fail-closed.

Privacy-safe retrieval telemetry

Every discover.js memory invocation appends aggregate metadata to ~/system/logs/discover-queries.jsonl (mode 0600). It records timestamp, query loglength/token wascount, foundrequested ontop-N, disk.result/confidence Thereforecounts, thislatency, baselinefail-closed isstate, aand labeledcoarse proxyerror forkind. missIt analysis,never not a real-traffic miss audit.

Recommendation: add privacy-awarerecords query loggingtext beforeor futurehashes, ranking/indexmemo experiments.filenames/content, secrets, or PII.

Back to top