Mail Pipeline E2E Debug — MC #105892 (2026-07-17)

Agent ID: john Verdict: FINDINGS (FINAL — svih 5 rola isporučilo; R5 E2E = FAIL na koracima slack-feed i search)

MC #105892 — Mail pipeline E2E debug: konsolidovani izvještaj

Tim: petter-graff (R1 arhitektura), martin-kleppmann (R2 fetch), codecraft (R3 triage), bruce-momjian (R4 storage/index), angie-jones (R5 E2E user test). Evidence: architecture.md, fetch-layer.md, triage-layer.md, index-layer.md, e2e-user-test.md (ovaj dir). Metod: read-only debug, bez izmjena koda, bez restarta daemona, LightRAG/neo4j netaknuti, 1 bounded test mail (R5).

⚠️ ISPRAVKA 2026-07-17 15:47 (post-fix verifikacija #105895)

Nalaz #1 ovog izvještaja ("TRIAGE MRTAV") je OBOREN živom re-verifikacijom u follow-up tasku #105895 (evidence ~/system/evidence/105895/fix-report.md): inbox-watcher launchd job je cijelo vrijeme bio zdrav i aktivno triažirao (runs=1919, exit 0, kontinuirana triaged_at progresija kroz 16–17.07; sintetički probe red procesiran 4m19s nakon inserta). "state=not running / active count=0" je normalno idle stanje StartInterval=300s joba između ciklusa — i R1/R3 i peer-verifier su to pogrešno interpretirali kao mrtav job. Watcher gejta na triaged_at, ne na status/classification kolone. Stvarni nađeni bugovi umjesto toga: (1) 2043 reda (2026-02-17 — 2026-04-26) trajno nedostupna triage upitu zbog 30-dnevnog prozora (inbox-watcher.js:679); (2) nema backlog guarda za buduće stvarne outage; (3) F4 kadenca boot-signala ostaje stvarni uzrok kašnjenja CEO signala. Follow-up F1 (#105895) preimenovan u nalaz+preporuke; nova follow-up potreba: orphan-window backfill s bounded/dry-run planom.

Izvršni sažetak (za CEO)

Mail sistem NIJE "slomljen na čitanju" — svih 19 naloga se polluje svakih 5 min i kanonski store (~/system/databases/email-inbox.db, 30.7 MB, 7518 mailova) se aktivno piše. Slomljeni su SORTIRANJE i INDEKSIRANJE, a MONITORING o fetch sloju sistematski laže:

1. TRIAGE MRTAV: inbox-watcher.js (jedina klasifikacijska autoritet, kreira MC taskove, označava \Seen) — njegov launchd job NIJE aktivan. Mailovi ulaze u DB i ostaju u status='new' zauvijek. Ovo je root cause zašto su Avaz/Asmir mailovi visili nepročitani 3 sedmice (uz sekundarni faktor: boot signal se izvršava samo na ručni session start, nema cron kadencu).
2. MONITORING LAŽE: email-imap-db-audit.js zna kredencijale za samo 5 od 18 naloga → 13 naloga "Unknown account" u svakom runu → 197 uzastopnih hourly lažnih/šumnih alarma od 10.07.; watchdog exit 1 je "alarm poslan" (by design), ne crash. Jedini TVRDI ingest gap: alai INBOX 69/70 poruka (zadnja 2 dana) nije ušlo u DB.
3. INDEKSIRANJE NE POSTOJI: discover.js nema EMAIL kategoriju; email-inbox.db nema FTS. 7518 mailova je nevidljivo tool-first pretrazi (gap protiv ZAKON NULA). Dodatno, 532/7518 (7.1%) redova nema body — backfill alat ima regex bug (0/96 success).
4. KRHKOST FETCHA: 19 sekvencijalnih bw get item poziva pod globalnim lockom po ciklusu → 1198 ETIMEDOUT u ~mjesec dana (samoizlječivo, ali hronični porez; 3 uzastopna faila = 30-min blackout po nalogu).
5. SECURITY: NODE_TLS_REJECT_UNAUTHORIZED=0 hardkodiran u plistu email-agenta (gasi TLS verifikaciju za cijeli proces) + copy-paste u vault-fix.sh i vault-keeper.js.

Ključne korekcije pogrešnih premisa (vrijednost cross-verifikacije)

• "email-inbox.db = 0 B mrtvi keš" → POGREŠNO: to je decoy u ~/system/tools/ (+ još 3 stub kopije); živa baza je ~/system/databases/email-inbox.db (R2+R4 nezavisno).
• "13/17 naloga ne fetcha" → POGREŠNO: to je bug audit alata (VAULT_NAMES 5/18), fetch radi (R2 enumeracija svih 19 + R3 root cause).
• "slack #inbox posta inbox-watcher svakih 5 min" → POGREŠNO: posta mail-digest.js, event-driven 8–100+ min (R3); watcher uopšte ne radi.
• "signal filter guta Avaz mailove" → POGREŠNO: grep pattern ispravan (repro R3); problem je kadenca izvršavanja + slice(-15) latentni rizik.
• himalaya putanja nije mrtav kod: namjerno HIMALAYA_DISABLED=1 (MC #104595, R2).

Detaljni nalazi po sloju

R1 architecture.md — dvofazna arhitektura (email-agent fetch → email-inbox.db → inbox-watcher triage), tabela kanonskih skripti vs duplikata/decoya, 8 rangiranih failure modova.

R2 fetch-layer.md — per-account tabela (19/19 OK u ciklusu 12:41–12:44), bw lock bottleneck mehanika (mail-native.js:117-210), circuit breaker putanja, TLS=0 lokacija (plist), himalaya status, DB dokaz da su Avaz/Asmir mailovi ingestovani 5–6 dana kasno (01.07. 19:42) u status='new'.

R3 triage-layer.md — ingest-monitor exit-kod semantika (0/1/2), VAULT_NAMES 5-vs-18 root cause, gap-state 197h, signal-filter repro (UID 51/52/53 matchuju), slack feed liveness (12:43), email-classifications.jsonl mrtav od 19.06. (writer nikad daemonizovan), DLQ replay path mismatch (~/Library/Logs/ALAI/ vs ~/system/logs/).

R4 index-layer.md — kanonski store dokaz (7518 redova, MAX created_at danas 12:44), discover.js bez EMAIL izvora (test upiti vraćaju samo LightRAG/BookStack ručne bilješke), nema FTS tabele, body-integrity 0/96 regex bug, alai INBOX 69/70 gap, audit "Unknown account" za 13 naloga.

R5 e2e-user-test.md — VERDIKT: FAIL (2 od 6 koraka pala). Test mail UID 647, poslan 13:01:30Z.

• SEND: PASS (uz 1 bw vault timeout od 30 s prije uspješnog retryja — potvrđuje R2 contention).
• UNREAD: PASS — pojavio se ~4 s nakon slanja, stabilan i nakon 11 min.
• SIGNAL FILTER: PASS — test mail ispravno NE matchuje, stvarni Avaz/Asmir signali i dalje matchuju (nula lažnih pozitiva/negativa — potvrđuje R3 repro).
• SLACK #inbox: FAIL — 3 polla kroz 11 min, feed zamrznut na 12:43; ni test mail ni drugi novi mail (UID 648) nisu objavljeni. Napomena sinteze: R3 je utvrdio da poster (mail-digest.js) objavljuje event-driven s razmacima 8–100+ min, pa je "FAIL unutar 11 min" FAIL naspram DOKUMENTOVANOG očekivanja od ~5 min — samo očekivanje je pogrešno/nedokumentovano, a feed je u najboljem slučaju nepredvidiv (za CEO signal svrhu = nepouzdan).
• READ: PASS — header/body/marker tačni.
• SEARCH (discover.js): FAIL — 0 stvarnih pogodaka u 2 pokušaja / 11 min. Napomena sinteze: Angin pripisani uzrok ("email-inbox.db = 0 B keš") je zastarjela premisa — stvarni uzrok po R4: discover.js nema EMAIL izvor + baza nema FTS. Zaključak (nema živog mail indeksa) stoji. Angine preporuke pokrivene follow-upovima F3 (search) i F4/F1 (feed/triage kadenca).

Follow-up fix taskovi (prijedlog, po prioritetu)

F1 (H) Reaktivirati/popraviti inbox-watcher launchd job — triage autoritet ne radi; mailovi ostaju u status='new'; uzrok utvrditi instrumentisanim replayem (logovi rotirani). F2 (H) email-imap-db-audit.js: proširiti VAULT_NAMES na svih 18 naloga → gasi 197h lažnih alarma i vraća povjerenje u monitoring; zatim istražiti stvarni alai INBOX 69/70 gap. F3 (M) discover.js EMAIL kategorija + FTS5 indeks nad emails(subject, body, from_addr) — zatvara ZAKON NULA gap. F4 (M) Boot-signal kadenca: launchd/cron trigger za signal grep (ne samo ručni session start); riješiti slice(-15) prozor za stare unread. F5 (M) email-body-integrity-check.js envelope-ID regex fix (0/96) + backfill 532 praznih bodyja. F6 (M) Vault throughput: keširati bw kredencijale po ciklusu / smanjiti 19 sekvencijalnih poziva pod lockom (1198 ETIMEDOUT/mjesec). F7 (L) Ukloniti NODE_TLS_REJECT_UNAUTHORIZED=0 iz plist/vault-fix.sh/vault-keeper.js — CA chain fix (Securion). F8 (L) DLQ replay path fix + odluka o email-classifications.jsonl writeru (daemonizovati ili ugasiti); počistiti 4 decoy email-inbox.db stuba i .bak fajlove.

Follow-up 2026-07-17 — MC #105896: VAULT_NAMES fix + alai INBOX gap root cause

DIO A: email-imap-db-audit.js VAULT_NAMES map extended from 5 to all 18 accounts (was causing 13x "Unknown account" per run, 197h of false alarms on #exec since 2026-07-10). Fixed using mail-native.js VAULT_NAMES as canonical source. Verified live: 0/18 errors, 73s wall time, well inside the 3600s hourly monitor budget.

DIO B — alai INBOX gap root cause (confirmed): alai account DB ingestion stalled since 2026-07-15 22:18:47 while john/gmail ingest live every cycle. Root cause: alai and john are two account labels in the daemon pointing at the identical mailbox (both map to vault item Migadu — [email protected]). Since Himalaya is disabled (MC #104595, HIMALAYA_DISABLED=1 live), LEGACY_IMAP_HOT_PATH=true, and the IMAP search criteria reverts to { unseen: true } instead of date-range — a regression of the bug MC #101887 already fixed (code comment at email-agent.js:765-766 documents the original fix). Because john's pass runs first each cycle and marks messages \Seen, alai's pass moments later finds nothing left to fetch. Confirmed with direct IMAP flag checks on the 10 most recent missing alai UIDs — all already \Seen.

Follow-up candidate (not implemented, out of #105896 scope): re-apply MC #101887's date-range + DB-dedup search regardless of LEGACY_IMAP_HOT_PATH, in email-agent.js:791.

Evidence: ~/system/evidence/105896/ (fix-report.md, audit-run2-stdout.log, imap-db-diff-run2.json, peer-verify.md, verdict.json). P2P peer verification: PASS, 6/6 checks, including independent live re-execution of the audit.