# Bilko Sentinel — Tier-1 Bounded Auto-Remediation (Shadow-First) 2026-06-11

## Status

BUILT — SHADOW mode. MC #103435 (AgentForge build) + MC #103436 (Securion adversarial review). Parent MC #103328. Module: `/Users/makinja/system/tools/bilko-sentinel-tier1.js`. **Tier-0 remains the active detection layer; Tier-1 is shadow-calibrating only.**

**Related:** [Bilko Sentinel Tier-0](/books/bilko-balkan-accounting-saas/page/bilko-sentinel-tier-0-self-healing-agent-2026-06-10) (detection layer, LIVE) — [Bilko Observability (GCP-native)](/books/bilko-balkan-accounting-saas/page/bilko-observability-gcp-native-2026-06-10)

SRE rationale: DECISIONS-observability-2026-06-10.md, Decision 2 — Kelsey Hightower consult. The bar is not yet met; ship the muscle, start the calibration clock, arm only when proven.

## What Tier-1 Is

Tier-1 is the agent that can **fix**. Where Tier-0 only detects, diagnoses, and proposes — Tier-1 computes the exact bounded action and, when armed, executes it. Permitted action set (exhaustive):

- Roll back `bilko-api-demo` or `bilko-web-demo` to N-1 only (never older)
- Set Cloud Run `--min-instances` 0→1 (warm floor for cold-start incidents)
- Slack escalation (always permitted, labelled with current mode)

**Never-automate** (enforced at IAM, not policy): no IAM/policy change, no Cloud SQL op, no secret op, no DNS/LB/network, no rollback older than N-1, no action during an in-flight deploy or protected business window. Invoked by the Tier-0 loop via try/catch — any Tier-1 error is caught and logged, Tier-0 continues unaffected.

## Modes (SENTINEL\_TIER1\_MODE)

Mode is read **once at startup**, immutable for the life of the process. Unrecognised or missing value resolves to `shadow`. The LaunchAgent plist deliberately omits the key.

<table id="bkmrk-modebehaviourcurrent"> <thead><tr><th>Mode</th><th>Behaviour</th><th>Current state</th></tr></thead> <tbody> <tr> <td>**shadow (DEFAULT)**</td> <td>On a breach: compute the bounded action, announce to #ceo with gate evaluation, write one row to calibration ledger. Execute nothing.</td> <td>ACTIVE</td> </tr> <tr> <td>`ack`</td> <td>Same as shadow, but if a human posts APPROVE in the #ceo thread within 3 min, execute the action (all 8 gates must pass). Slack poll loop is a follow-on — currently defers conservatively. Requires F4 hardening first.</td> <td>NOT YET WIRED</td> </tr> <tr> <td>`auto`</td> <td>Execute automatically when all 8 gates pass and promotionBarMet() returns true. Silence in ack window = proceed. FORBIDDEN until bar met + human-engineer sign-off.</td> <td>BLOCKED (PROMOTION\_BAR\_NOT\_MET)</td> </tr> </tbody></table>

## Current State: SHADOW — Confirmed Inert

Securion adversarial review (MC #103436) + AgentForge build verification (MC #103435) independently confirm: **in shadow mode this agent cannot mutate prod.** Two independent structural barriers:

1. `handleIncident()` enters an `if (MODE === 'shadow')` block and **returns at line 852** — the execution block at line 861+ is outside and structurally unreachable.
2. `executeRollback()` (line 625) and `executeScaleFloor()` (line 675) each **throw as their literal first statement** when MODE === 'shadow'. Both barriers are independently sufficient.

Live production traffic verified unchanged during shadow simulation: `bilko-api-demo-00192-sfv @ 100%` before and after. Auto mode tested with `SENTINEL_TIER1_MODE=auto` and hard-blocked with PROMOTION\_BAR\_NOT\_MET.

## 8 Pre-fire Gates

ALL eight must be true before any execution in ack or auto mode. In shadow they are evaluated and recorded to the ledger only.

<table id="bkmrk-%23gatedetail-1alert-s"> <thead><tr><th>\#</th><th>Gate</th><th>Detail</th></tr></thead> <tbody> <tr><td>1</td><td>Alert sustained ≥5 min</td><td>Prevents action on transient spikes. Measured from `incident.firstSeenAt`.</td></tr> <tr><td>2</td><td>Calibrated LLM confidence</td><td>Requires "high" until ≥5 ledger reviews; adjusts to "medium" with ledger data. Derived from calibration, not hardcoded.</td></tr> <tr><td>3</td><td>N-1 confirmed healthy ≥10 min</td><td>Rollback target must have been in Ready=True state for ≥10 min before the current (bad) revision. Unknown = block + escalate.</td></tr> <tr><td>4</td><td>No schema migration in bad revision</td><td>Requires deploy manifest (`~/system/state/bilko-deploy-manifest.json`). **Honest block: absent manifest = BLOCK + escalate to human.** Rolling back across a schema migration can corrupt data.</td></tr> <tr><td>5</td><td>Cooldown: no action in last 60 min</td><td>One action per 60-minute window across all types.</td></tr> <tr><td>6</td><td>3-min human-ack window</td><td>HOLD or ABORT in #ceo thread cancels. Shadow: informational. Ack: requires explicit APPROVE. Auto: silence = proceed.</td></tr> <tr><td>7</td><td>IAM diff vs known-good snapshot</td><td>Compares live Cloud Run service IAM policy against `~/system/state/bilko-sentinel-iam-snapshot.json`. Mismatch = block + escalate. Motivated by the 2026-06 IAM wipe incident.</td></tr> <tr><td>8</td><td>N-1 is not itself a rollback revision</td><td>Prevents rolling back to a revision tagged as a known-bad rollback. Checked against deploy manifest `isRollback` flag.</td></tr> </tbody></table>

## Circuit Breakers

- **Max 2 actions / 24h** across all types. Third incident in 24h → human-only escalation.
- **Self-disable after failed remediation:** post-action health check at +5 min; if service still unhealthy → `circuitOpen=true`, SENTINEL-CIRCUIT-OPEN to Slack + email. Manual re-enable: set `circuitOpen=false` in `~/system/state/bilko-sentinel-tier1-state.json`.
- **Single-writer lock:** atomic file lock (`~/system/state/bilko-sentinel-tier1.lock`) — prevents race on same revision.
- **Audit-before-execute:** `~/system/logs/bilko-sentinel-audit.jsonl` written before any gcloud mutation verb. Log write failure → action does not fire.
- **Two Slack announcements per action:** BEFORE ("about to roll back X to rev Y in 3 min unless HOLD") and AFTER ("rolled back, health check in 5 min").

## Promotion Bar: shadow → auto

`promotionBarMet()` is a hard gate on the auto path. Reads the calibration ledger at runtime — not hardcoded. Currently returns **FALSE**; auto path refuses with PROMOTION\_BAR\_NOT\_MET.

<table id="bkmrk-criterionrequiredcur"> <thead><tr><th>Criterion</th><th>Required</th><th>Current (2026-06-11)</th></tr></thead> <tbody> <tr><td>daysLive\_gte30</td><td>≥30 days since first ledger entry</td><td>0 days — NOT MET</td></tr> <tr><td>evaluatedProposals\_gte20</td><td>≥20 proposals in ledger</td><td>2 — NOT MET</td></tr> <tr><td>fpRate\_lt5pct</td><td>Human-reviewed FP rate &lt;5%</td><td>100% default — NOT MET</td></tr> <tr><td>groundTruthHit</td><td>≥1 row with human\_verdict=correct</td><td>0 — NOT MET</td></tr> <tr><td>deployManifestExists</td><td>~/system/state/bilko-deploy-manifest.json present</td><td>Absent — NOT MET</td></tr> </tbody></table>

When the bar is eventually met, **human-engineer sign-off is still required** before the plist is updated to set `SENTINEL_TIER1_MODE=auto`. That is an explicit audited step, not an automatic promotion.

## Arming Prerequisites (Securion #103436 — MC #103439)

These gate **arming (ack/auto), not shadow.** Securion re-review required before the mode key is added to the plist.

<table id="bkmrk-findingseverityrequi"> <thead><tr><th>Finding</th><th>Severity</th><th>Required before arming</th></tr></thead> <tbody> <tr> <td>**F5 — Ledger integrity**</td> <td>HIGH</td> <td>HMAC-sign each ledger row (key stored outside ledger path). promotionBarMet() must verify HMAC before counting. Without this, forged human\_verdict entries can satisfy the promotion bar and arm auto.</td> </tr> <tr> <td>**F7 — SA IAM scope**</td> <td>MEDIUM</td> <td>Verify alai-cli-deployer holds only monitoring.viewer + logging.viewer + run.viewer in shadow. For auto: run.developer scoped by resource condition to bilko-api-demo + bilko-web-demo only. Must NOT hold cloudsql.\*, iam.\*, secretmanager.\*, dns.\*.</td> </tr> <tr> <td>**F4 — Ack approver allowlist**</td> <td>INFO</td> <td>Before Slack poll loop is wired: define constant with allowed Slack user IDs. Poll must verify message.user + thread\_ts — not just message text.</td> </tr> <tr> <td>**F6 — IAM snapshot seal**</td> <td>MEDIUM</td> <td>chmod 0444 after first write. Add sealed flag; require manual unsealing for reset. Populate bilko-web-demo immediately.</td> </tr> <tr> <td>**F2 — Misleading Object.freeze**</td> <td>MEDIUM</td> <td>Remove Object.freeze({MODE}) at line 57 — it freezes a discarded object, not the const binding. Replace with clarifying comment.</td> </tr> <tr> <td>**F8 — Gate 8 inconsistency**</td> <td>LOW</td> <td>Align Gate 8 with Gate 4: block (not warn-and-pass) when deploy manifest absent or N-1 not in manifest.</td> </tr> <tr> <td>**F3 — Module integrity**</td> <td>LOW</td> <td>Add startup SHA-256 check of the module file against a stored known-good value outside the module path.</td> </tr> </tbody></table>

All items tracked in MC #103439. **Securion re-review required before mode key is added to plist.**

## Calibration Ledger

Every shadow proposal appends one row to `/Users/makinja/system/logs/bilko-sentinel-tier1-ledger.jsonl`

Row schema: `{ ts, incidentId, policyName, condName, resource, diagnosis, confidence, computedAction, n1Info, gates, mode, human_verdict }`

The `human_verdict` field starts as `"not-yet-reviewed"`. Update to: `correct` | `wrong-rootcause` | `would-have-worsened`. A **weekly summary** is posted to #ceo automatically: proposal count, reviewed count, FP rate, ground-truth hits, promotion bar status.

```
node /Users/makinja/system/tools/bilko-sentinel-tier1.js --weekly-summary
```

## Infrastructure

<table id="bkmrk-componentlocation-mo"> <thead><tr><th>Component</th><th>Location</th></tr></thead> <tbody> <tr><td>Module</td><td>`/Users/makinja/system/tools/bilko-sentinel-tier1.js`</td></tr> <tr><td>Weekly summary plist</td><td>`/Users/makinja/system/tools/com.alai.bilko-sentinel-tier1-weekly-summary.plist`</td></tr> <tr><td>Calibration ledger</td><td>`/Users/makinja/system/logs/bilko-sentinel-tier1-ledger.jsonl`</td></tr> <tr><td>IAM snapshot</td><td>`/Users/makinja/system/state/bilko-sentinel-iam-snapshot.json`</td></tr> <tr><td>State file</td><td>`/Users/makinja/system/state/bilko-sentinel-tier1-state.json`</td></tr> <tr><td>Execution audit log</td><td>`/Users/makinja/system/logs/bilko-sentinel-audit.jsonl`</td></tr> <tr><td>Run log</td><td>`/Users/makinja/system/logs/bilko-sentinel-tier1.log`</td></tr> <tr><td>Single-writer lock</td><td>`/Users/makinja/system/state/bilko-sentinel-tier1.lock`</td></tr> <tr><td>GCP project</td><td>`tribal-sign-487920-k0`, region `europe-north1`</td></tr> <tr><td>SA</td><td>`alai-cli-deployer@tribal-sign-487920-k0.iam.gserviceaccount.com`</td></tr> <tr><td>Allowed services</td><td>`bilko-api-demo`, `bilko-web-demo`</td></tr> <tr><td>Host</td><td>ANVIL (makinja local Mac) — invoked by Tier-0 loop</td></tr> </tbody></table>

## Runbook

### Self-test in shadow

```
node /Users/makinja/system/tools/bilko-sentinel-tier1.js --self-test
```

### Evaluate promotion bar

```
node /Users/makinja/system/tools/bilko-sentinel-tier1.js --promotionbar-test
# exits 0 if bar met, 1 if not
```

### Inspect calibration ledger

```
tail -f /Users/makinja/system/logs/bilko-sentinel-tier1-ledger.jsonl
```

### Re-enable after circuit-open

```
# Edit ~/system/state/bilko-sentinel-tier1-state.json
# Set "circuitOpen": false
```

### Flip to ack mode (AFTER all hardening items + Securion re-review)

```
# Add to LaunchAgent plist EnvironmentVariables:
#   <key>SENTINEL_TIER1_MODE</key><string>ack</string>
launchctl unload ~/Library/LaunchAgents/com.alai.bilko-sentinel.plist
launchctl load ~/Library/LaunchAgents/com.alai.bilko-sentinel.plist
```