Bilko Sentinel — Tier-1 Bounded Auto-Remediation (Shadow-First) 2026-06-11

Status

BUILT — SHADOW mode. MC #103435 (AgentForge build) + MC #103436 (Securion adversarial review). Parent MC #103328. Module: /Users/makinja/system/tools/bilko-sentinel-tier1.js. Tier-0 remains the active detection layer; Tier-1 is shadow-calibrating only.

Related: Bilko Sentinel Tier-0 (detection layer, LIVE) — Bilko Observability (GCP-native)

SRE rationale: DECISIONS-observability-2026-06-10.md, Decision 2 — Kelsey Hightower consult. The bar is not yet met; ship the muscle, start the calibration clock, arm only when proven.

What Tier-1 Is

Tier-1 is the agent that can fix. Where Tier-0 only detects, diagnoses, and proposes — Tier-1 computes the exact bounded action and, when armed, executes it. Permitted action set (exhaustive):

• Roll back bilko-api-demo or bilko-web-demo to N-1 only (never older)
• Set Cloud Run --min-instances 0→1 (warm floor for cold-start incidents)
• Slack escalation (always permitted, labelled with current mode)

Never-automate (enforced at IAM, not policy): no IAM/policy change, no Cloud SQL op, no secret op, no DNS/LB/network, no rollback older than N-1, no action during an in-flight deploy or protected business window. Invoked by the Tier-0 loop via try/catch — any Tier-1 error is caught and logged, Tier-0 continues unaffected.

Modes (SENTINEL_TIER1_MODE)

Mode is read once at startup, immutable for the life of the process. Unrecognised or missing value resolves to shadow. The LaunchAgent plist deliberately omits the key.

Current State: SHADOW — Confirmed Inert

Securion adversarial review (MC #103436) + AgentForge build verification (MC #103435) independently confirm: in shadow mode this agent cannot mutate prod. Two independent structural barriers:

1. handleIncident() enters an if (MODE === 'shadow') block and returns at line 852 — the execution block at line 861+ is outside and structurally unreachable.
2. executeRollback() (line 625) and executeScaleFloor() (line 675) each throw as their literal first statement when MODE === 'shadow'. Both barriers are independently sufficient.

Live production traffic verified unchanged during shadow simulation: bilko-api-demo-00192-sfv @ 100% before and after. Auto mode tested with SENTINEL_TIER1_MODE=auto and hard-blocked with PROMOTION_BAR_NOT_MET.

8 Pre-fire Gates

ALL eight must be true before any execution in ack or auto mode. In shadow they are evaluated and recorded to the ledger only.

Circuit Breakers

• Max 2 actions / 24h across all types. Third incident in 24h → human-only escalation.
• Self-disable after failed remediation: post-action health check at +5 min; if service still unhealthy → circuitOpen=true, SENTINEL-CIRCUIT-OPEN to Slack + email. Manual re-enable: set circuitOpen=false in ~/system/state/bilko-sentinel-tier1-state.json.
• Single-writer lock: atomic file lock (~/system/state/bilko-sentinel-tier1.lock) — prevents race on same revision.
• Audit-before-execute: ~/system/logs/bilko-sentinel-audit.jsonl written before any gcloud mutation verb. Log write failure → action does not fire.
• Two Slack announcements per action: BEFORE ("about to roll back X to rev Y in 3 min unless HOLD") and AFTER ("rolled back, health check in 5 min").

Promotion Bar: shadow → auto

promotionBarMet() is a hard gate on the auto path. Reads the calibration ledger at runtime — not hardcoded. Currently returns FALSE; auto path refuses with PROMOTION_BAR_NOT_MET.

When the bar is eventually met, human-engineer sign-off is still required before the plist is updated to set SENTINEL_TIER1_MODE=auto. That is an explicit audited step, not an automatic promotion.

Arming Prerequisites (Securion #103436 — MC #103439)

These gate arming (ack/auto), not shadow. Securion re-review required before the mode key is added to the plist.

All items tracked in MC #103439. Securion re-review required before mode key is added to plist.

Calibration Ledger

Every shadow proposal appends one row to /Users/makinja/system/logs/bilko-sentinel-tier1-ledger.jsonl

Row schema: { ts, incidentId, policyName, condName, resource, diagnosis, confidence, computedAction, n1Info, gates, mode, human_verdict }

The human_verdict field starts as "not-yet-reviewed". Update to: correct | wrong-rootcause | would-have-worsened. A weekly summary is posted to #ceo automatically: proposal count, reviewed count, FP rate, ground-truth hits, promotion bar status.

node /Users/makinja/system/tools/bilko-sentinel-tier1.js --weekly-summary

Infrastructure

Runbook

Self-test in shadow

node /Users/makinja/system/tools/bilko-sentinel-tier1.js --self-test

Evaluate promotion bar

node /Users/makinja/system/tools/bilko-sentinel-tier1.js --promotionbar-test
# exits 0 if bar met, 1 if not

Inspect calibration ledger

tail -f /Users/makinja/system/logs/bilko-sentinel-tier1-ledger.jsonl

Re-enable after circuit-open

# Edit ~/system/state/bilko-sentinel-tier1-state.json
# Set "circuitOpen": false

Flip to ack mode (AFTER all hardening items + Securion re-review)

# Add to LaunchAgent plist EnvironmentVariables:
#   <key>SENTINEL_TIER1_MODE</key><string>ack</string>
launchctl unload ~/Library/LaunchAgents/com.alai.bilko-sentinel.plist
launchctl load ~/Library/LaunchAgents/com.alai.bilko-sentinel.plist