Skip to main content

John+AI Factory Unified Fix - 2026-05-17 Session

John + AI Factory Unified Fix — 2026-05-17 Session

Date: 2026-05-17
Session ID: (recorded in session-state.md)
Lead Architect: Petter Graff (CodeCraft)
Root Cause Document: ~/system/specs/john-ai-factory-unified-fix-2026-05-17.md
Parent: Reality Anchor Doctrine v1 Final

Overview

This session converged two parallel problems into a single unified fix:

  1. John's hallucination defects (6 incidents in May 2026 alone)
  2. AI Factory structural gaps (RAG queue 3,150 items, Opus $9,790/day burn, edita dead-letter 161 tasks)

Petter Graff: "John is not a user of the AI Factory — John is the orchestration layer of the AI Factory, which means John's hallucination defects and the factory's structural gaps are the same problem seen from two angles."

Root Cause (Petter Panel Diagnosis)

The 52 rules and 11 hooks all share one fatal flaw: they are evaluated by the same LLM system they are meant to constrain.

When John claims "4/4 logins working" (Bilko UAT 2026-05-16), no deterministic probe ran. John synthesized a prose assertion from subagent output, and the gate accepted the file's existence as proof of its content.

This is the Writer = Witness antipattern compounded by a deeper epistemological error: rules written in natural language are interpreted by an LLM under execution pressure, and under pressure LLMs compress uncertainty into confident-sounding summaries.

More rules do not fix this. The attack surface is not insufficient rules — it is that the enforcement mechanism is the same substrate as the offender.

Structural Fixes Shipped (2026-05-17)

1. Opus Cost Guard Hook (MC #101140)

Problem: $9,790/day Opus burn on routine specialist dispatches (ALAI revenue = $0)
Fix: PreToolUse hook blocks Opus model on codecraft/vizu/proveo/flowforge/etc. Allows Opus only for novel architecture personas (petter-graff, martin-kleppmann) and /prompt-forge dispatches.
Impact: Projected $9,790/day → $500/day (~$278,700/month savings)

Documentation: Opus Cost Guard Hook

2. Claim Schema Injector (MC #101065)

Problem: No claim template pre-registered at task dispatch — verifier fills from John prose instead of probe output.
Fix: mc.js start fires schema-injector.js → writes /tmp/claim-schema-<id>.json with PENDING stubs. Verifier MUST fill stubs from deterministic probe output. Schema-stub-gate.sh blocks mc.js ready/done if any stub remains PENDING/FAILED.
Impact: Closes evidence padding attack surface (Bilko UAT incident root cause)

Documentation: Schema Stub Gate + Claim Schema Injector

3. Force Approval Queue (MC #100818 — Reality Anchor P1.1)

Problem: mc.js done --force allowed agents to bypass evidence gates immediately.
Fix: --force no longer executes immediately. Enqueues to ~/system/state/force-pending.jsonl with 24h TTL. CEO must approve via mc.js force-approve <queue_id> or deny via mc.js force-deny. Auto-expires after 24h.
Impact: Removes structural bypass; CEO-only gate override

Documentation: mc.js Force Approval Queue

4. Four Deterministic Probes (MCs #101133–#101136)

Problem: No deterministic probe framework — all evidence was LLM-narrated prose.
Fix: 4 probes shipped with registry at ~/system/probes/registry.json:

  • login-probe.sh — login verification (claim_class: login_works)
  • git-diff-probe.sh — commit verification (claim_class: commit_verified)
  • playwright-a11y-probe.js — a11y violation count (claim_class: a11y_count)
  • test-enumeration.sh — test case enumeration (claim_class: test_count)

Each probe outputs structured JSON with cryptographic seal. Probe output IS the evidence; LLM removed from evidence chain.

Documentation: 4 Deterministic Probes

5. Attack J Security Fix (MC #101149)

Problem: Evidence-ledger writer identity could be spoofed via --actor CLI parameter, bypassing Writer ≠ Witness gate.
Fix: Remove || actor from identity fallback chain (lines 2843, 3538, 3574, 3589 in mc.js). Agent identity MUST come from CLAUDE_AGENT_ID environment variable only (runtime-provided, not user-supplied).
Impact: Closes privilege escalation via identity forgery. Proveo verdict PARTIAL → PASS.

Documentation: Attack J Security Fix

AI Factory Top-3 Priorities (Petter Analysis)

Priority 1: RAG Drain-Worker (3,150 items blocked) ✅ DONE

Problem: RAG queue stalled on Vaultwarden CF Access timeout. Every agent operating on weeks-stale knowledge base.
Fix: Credential refresh + queue drain + live depth monitor wired.
Impact: Knowledge base current; reduces agent hallucination on system state.

Priority 2: Opus Cost Guard ✅ DONE

Problem: $9,790/day burn (zero revenue startup).
Fix: Hook shipped (see above).
Impact: Runway extended ~9 months.

Priority 3: Edita Dead-Letter Queue (161 tasks) — PENDING

Problem: 161 automation chains silently failed; unknown termination state.
Status: Triage pending (follow-up MC required).
Impact: Data integrity — cannot measure factory output accurately while 161 tasks have unknown state.

Convergence Principle

Petter Graff: "A 'fixed John' that runs deterministic probes before closing tasks directly demands a factory that can produce probe output on demand: the RAG pipeline must be current so probes have accurate baseline state, the edita queue must be drained so task completion signals are trustworthy, and the model routing must be governed so the orchestrator operates within budget constraints."

The unified system:

  • Deterministic observation (probes, not LLM prose)
  • LLM orchestration (routing, reasoning, delegation)
  • Structural gates between them (schema-stub-gate, force-approval-queue, opus-cost-guard)

The LLM stays in the chain for reasoning and routing. It exits the chain entirely for evidence production.

MCs Delivered

MC Title Status
#101140 Opus cost guard hook DONE
#101065 Deterministic session compiler (expanded scope) DONE
#100818 Reality Anchor P1.1 — force approval queue DONE
#101133 Probe: login-probe.sh DONE
#101134 Probe: git-diff-probe.sh DONE
#101135 Probe: playwright-a11y-probe.js DONE
#101136 Probe: test-enumeration.sh DONE
#101149 Attack J security fix DONE

Open Follow-Ups

  • INV1 + fork gap (MC #100825): Commit manifest as first-class evidence for any code-touch task
  • Tamper audit.log (MC #100823): Content-addressed audit ledger
  • qa-19 inputs (MC #100827): Verifier input validation
  • Playwright npm install: cd ~/system/probes && npm install && npx playwright install chromium
  • lightrag-migrate-pump: Backfill pre-May sessions into RAG
  • RAG dead-letter triage: Review 3,150 drained items for loss
  • Edita dead-letter queue: Triage 161 tasks (Priority 3)

Where to Read More

Memory Snapshot

Full session details archived to:
~/.claude/projects/-Users-makinja/memory/project_john_factory_unified_fix_2026-05-17.md

This page is the umbrella documentation for the 2026-05-17 unified fix session. All 5 component pages are linked above.

No comments to display

Back to top