Skip to main content

Memory Retrieval Eval Harness

Memory Retrieval Eval Harness — MC #105224

Purpose

Regression-test discover.js memory "<query>" retrieval quality with a labeled set of real incident queries mapped to known memo files.

Artifacts

  • Harness: ~/system/tools/memory-eval-harness.js
  • Test set: ~/system/specs/memory-retrieval-eval/test-set-v1.json
  • Query-log audit: ~/system/specs/memory-retrieval-eval/query-log-miss-audit-2026-07-10.md
  • Fresh evidence: ~/system/evidence/memory-eval-105224-verify-20260710/

How to run

node ~/system/tools/memory-eval-harness.js run \
  --test-set ~/system/specs/memory-retrieval-eval/test-set-v1.json \
  --output ~/system/evidence/memory-eval-$(date +%Y%m%d-%H%M%S)

Dry-run schema validation:

node ~/system/tools/memory-eval-harness.js run \
  --test-set ~/system/specs/memory-retrieval-eval/test-set-v1.json \
  --dry-run

Metrics

  • Recall@3: correct memo appears in the top 3 files returned by discover.js memory.
  • MRR: reciprocal rank of the correct memo; 0 when not returned.

Important limitation: discover.js memory hard-caps output at top 3, so this harness cannot distinguish rank 4+ from a complete miss.

Baseline run — 2026-07-10

Fresh verification output: ~/system/evidence/memory-eval-105224-verify-20260710/

  • Labeled pairs: 40
  • Recall@3: 0.600 (24/40)
  • MRR: 0.546
  • Errors: 0/40
  • p50 latency: 174ms
  • p95 latency: 243ms

Query-log audit

No persistent historical discover.js query log was found on disk. Therefore this baseline is a labeled proxy for miss analysis, not a real-traffic miss audit.

Recommendation: add privacy-aware query logging before future ranking/index experiments.

Back to top