Skip to main content

mini-verifier canary — dnevni known-answer monitoring (MC #105136)

MC #105136 — mini-verifier canary in auto-verify-regression — Canary Report

Scope

Extended ~/system/tools/auto-verify-regression.js with a new "mini-verifier canary" section (3 known-answer cases run daily through ~/system/tools/mini-verifier.js, the pi-orch $0/call local-LLM closure gate). Approved by CEO 2026-07-09 as a 30-day monitoring pilot (#105107/#105113).

Files touched

  • Edited: /Users/makinja/system/tools/auto-verify-regression.js (only file edited; no new LaunchAgent, no edits to mini-verifier.js, no edits to the existing 5 claim-verifier TESTS)
  • New (evidence only, not production): /Users/makinja/system/evidence/105136/*
  • Production monitoring feed: /Users/makinja/system/state/mini-verifier-canary.jsonl

AC Checklist

# Acceptance criterion Result
1 New "mini-verifier canary" section added after the existing 5 slučajeva PASS — CANARY_CASES array + runCanaryCase/runCanarySection defined after TESTS/runTest, invoked in main() after the claim-verifier loop
2 C1: true claim about existing file (mc.js) with that file as evidence → expect CONFIRMED PASS — verdict CONFIRMED, matched
3 C2: claim about nonexistent file + same nonexistent path as evidence → expect HALLUCINATION PASS — verdict HALLUCINATION, matched
4 C3: claim about nonexistent file + unrelated existing evidence (BUILD-BLUEPRINT.md, #105113 repro) → expect HALLUCINATION PASS — verdict HALLUCINATION, matched
5 Mismatch or SKIP on C1-C3 → same Slack alert mechanism reused PASS (verified in V2) — reuses existing sendSlackAlert() → slack.js send ai-ops; SKIP does NOT alert (treated as WARN per spec), only true verdict mismatch alerts
6 Every run appends JSON line to ~/system/state/mini-verifier-canary.jsonl (ts, case, expected, actual, reasons, latency_ms, model_id) PASS — all 6 fields present, verified in V1 output
7 Canary section does not break the 5 existing claim-verifier cases or change their exit-code semantics PASS — 5/5 claim-verifier PASS unaffected in both V1 and V2 runs; allPassed (claim-verifier) logic untouched, only OR'd with new canaryHasFail for overall exit code
8 No new LaunchAgent PASS — only pre-existing com.john.auto-verify-regression.plist present, unchanged, same script path (V3)
9 No edits to mini-verifier.js PASS — file untouched (only read for interface discovery)
10 New files TS-only N/A — no new production files were created; the only new "files" are evidence artifacts (backup .js copy, .txt outputs), not new production source files. The one edit is to the existing .js production file per explicit permission ("Editovanje postojećeg .js dozvoljeno")
11 No secrets in output PASS — grep scan of all evidence files for api-key/secret/token/password/bearer patterns returned nothing

V1 — Full manual run (production script, unmodified expectations)

Command: node ~/system/tools/auto-verify-regression.js --verbose Exit code: 0 Result: 5/5 claim-verifier PASS + 3/3 canary OK Raw stdout: V1-full-run-stdout.txt (stderr empty: V1-full-run-stderr.txt) jsonl after run (3 new lines, all status=OK): V1-canary-jsonl-after-run.txt

AUTO-VERIFY REGRESSION: 5/5 PASS
  v Tunnel count mismatch -> correctly FAIL
  v Domain existence false negative -> correctly FAIL
  v Email account count -> correctly FAIL
  v Live domain status -> correctly PASS
  v Domain exists -> correctly PASS
MINI-VERIFIER CANARY: 3/3 OK
  v mini-verifier canary: true claim, existing evidence -> correctly CONFIRMED
  v mini-verifier canary: hallucinated claim, nonexistent evidence -> correctly HALLUCINATION
  v mini-verifier canary: hallucinated claim, unrelated existing evidence (#105113 repro) -> correctly HALLUCINATION

V2 — Alert path validation (TEMPORARY test copy only, production untouched)

A temporary copy auto-verify-regression.TEST-COPY.js was made in the evidence dir (never in ~/system/tools/). In that copy ONLY: C1's expected was flipped from CONFIRMED to HALLUCINATION (intentionally wrong, to force a mismatch), and sendSlackAlert() was given a CANARY_TEST_DRY_RUN=1 env-gated dry-run branch that logs the exact command instead of executing it — so the real #ai-ops channel was NOT spammed with a synthetic test alert.

Command: CANARY_TEST_DRY_RUN=1 node .../auto-verify-regression.TEST-COPY.js --verbose Exit code: 1 (as expected — proves overall exit code goes non-zero on canary FAIL) Result: 5/5 claim-verifier PASS (unaffected) + 2/3 canary OK, 1 FAIL (C1, forced) Raw stdout: V2-alert-path-stdout.txt

Key excerpt proving the alert branch fired:

WARN:   FAIL: mini-verifier canary: true claim, existing evidence — got CONFIRMED, expected HALLUCINATION
MINI-VERIFIER CANARY: 2/3 OK
  x mini-verifier canary: true claim, existing evidence -> UNEXPECTED: got CONFIRMED, expected HALLUCINATION
INFO: [DRY-RUN] would exec: node .../system/tools/slack.js send ai-ops "[ALERT] mini-verifier canary FAILED: mismatch on known-answer case(s)
Failures:
- mini-verifier canary: true claim, existing evidence: got CONFIRMED, expected HALLUCINATION
mini-verifier.js may be broken. Check: /Users/makinja/system/state/mini-verifier-canary.jsonl"

This confirms: (a) the exact same sendSlackAlert() → slack.js send ai-ops call path used by the 5 existing claim-verifier cases is reused for canary failures, (b) the overall script exit code becomes 1 when a canary case fails even if all 5 claim-verifier cases still pass.

Cleanup honesty note: because CANARY_JSONL path was not overridden in the test copy, this V2 run also appended 3 lines (including one synthetic FAIL for C1) to the real production ~/system/state/mini-verifier-canary.jsonl. This was caught and corrected: the full polluted file was saved to V2-canary-jsonl-BEFORE-cleanup.txt for the record, then the production jsonl was restored to contain only the 3 genuine V1 lines (verified via diff against V1-canary-jsonl-after-run.txt before truncating). The 30-day monitoring feed is clean of test artifacts as of this report.

V3 — LaunchAgent unchanged

~/Library/LaunchAgents/com.john.auto-verify-regression.plist inspected: still the only relevant plist, ProgramArguments still points at /Users/makinja/system/tools/auto-verify-regression.js (same path, untouched schedule StartInterval=900, RunAtLoad=true). No new LaunchAgent files present. Saved to V3-launchagent-plist.txt.

Diff summary

diff-pre-vs-post.txt — 251 diff lines, all additive:

  • Updated file header comment to mention the canary section.
  • Added MINI_VERIFIER and CANARY_JSONL path constants.
  • Added CANARY_CASES array (C1/C2/C3), runCanaryCase(), appendCanaryJsonl(), runCanarySection().
  • In main(): after the existing claim-verifier report/log block, added a call to runCanarySection(), a canary report print block, and extended the alert condition to !allPassed || canaryHasFail (was !allPassed only) with an appended canary-specific alert message segment when applicable.
  • Zero lines removed/modified inside the original TESTS array, runTest(), or sendSlackAlert() in the production file (the dry-run branch only exists in the disposable TEST-COPY, not in production).

What was NOT done (explicitly out of scope, confirmed absent)

  • No new LaunchAgent plist created.
  • No edits to mini-verifier.js.
  • No edits to the 5 existing claim-verifier test cases or their pass/fail logic.
  • No secrets printed to stdout/stderr/evidence files (checked).

Evidence paths (all absolute)

  • /Users/makinja/system/evidence/105136/auto-verify-regression.js.pre-105136 (pre-edit backup)
  • /Users/makinja/system/evidence/105136/diff-pre-vs-post.txt
  • /Users/makinja/system/evidence/105136/V1-full-run-stdout.txt, V1-full-run-stderr.txt, V1-exit-code.txt
  • /Users/makinja/system/evidence/105136/V1-canary-jsonl-after-run.txt
  • /Users/makinja/system/evidence/105136/auto-verify-regression.TEST-COPY.js (disposable, alert-path test only)
  • /Users/makinja/system/evidence/105136/V2-alert-path-stdout.txt, V2-alert-path-stderr.txt, V2-exit-code.txt
  • /Users/makinja/system/evidence/105136/V2-canary-jsonl-BEFORE-cleanup.txt (honest record of the test-copy jsonl pollution, since corrected in production)
  • /Users/makinja/system/evidence/105136/V3-launchagent-plist.txt
  • /Users/makinja/system/evidence/105136/canary-report.md (this file)
  • Production: /Users/makinja/system/tools/auto-verify-regression.js (edited)
  • Production: /Users/makinja/system/state/mini-verifier-canary.jsonl (clean, 3 genuine lines from V1)

Evidence: ~/system/evidence/105136/ | Monitoring feed: ~/system/state/mini-verifier-canary.jsonl | Kadenca: svakih 15 min (StartInterval=900, com.john.auto-verify-regression) | Lanac: #105018 → #105104 → #105107 → #105113 → #105136 | P2P mesh: mesh-thr-439d874c / mesh-msg-2924726d | 2026-07-09

No comments to display

Back to top