From 2026-05-13 onward, ALAI runs **6+ concurrent Claude Code sessions daily** (12 sessions on 2026-05-15). Each session writes to shared state files with zero locking. On 2026-05-18 at 14:42, `~/system/memory/SESSION-STATE.md` was rewritten mid-session from session `256da42c` to session `a10b7bc9` **between two reads in the same `/sync` skill invocation** — John's continuity context silently flipped to another session's "Next Steps."
Three CEO-visible collisions confirmed before probing began: 1. **Session continuity lost** — John's "Next Steps" overwritten by last-writer-wins across concurrent sessions 2. **Gate verdicts corrupted** — `last-validator-verdict.json` written by session A, read by session B's `mc.js done`, passing/failing the wrong task 3. **Cost tracking undercount** — 1 of 4 concurrent Stop hooks' INSERTs lost in `costs.db`, causing `cost-tracker.js summary` to understate spend
The multi-session concurrency rate is accelerating: 6 sessions/day in May 2026 is 3× the February baseline. Without isolation, the collision surface grows quadratically.
---
## Collision Ledger
Empirical probe evidence from `/tmp/session-collision-20260518T{143721,143735}/probe.jsonl` (T3 Phase 1):
Each session writes to `-.` instead of a single global file. At session boot (P0-1 only), compaction merges all per-session files with mtime ≤ 4h into canonical view.
**Implementation:** - P0-1: `SESSION-STATE-.md` written by `session-ledger.sh`; compacted by `enforce-next-steps.sh` at boot (lines 62-108); cleanup in `parent-session-cleanup.sh` (line 74) - P0-2: `last-validator-verdict-.json` written by `session-output-validator.sh` (lines 491, 549); `mc.js done` reads per-session path (lines 2939-2966) with fail-closed gate if absent - P0-5: `/tmp/incident-mode-` written by `incident-response-mode.sh` (lines 31-42); orphan purge at 4h (lines 52-59) - P0-6: `/tmp/prompt-forge-active-` set by `/prompt-forge` skill (SKILL.md Step 0, line 57); reader bypass in `sonnet-default-gate.sh` (line 108) and `claude-sonnet-default.sh` (line 16)
**Rollback:** Set `ISOLATION_SESSION_STATE_SCOPED=0`, `ISOLATION_VERDICT_SESSION_SCOPE=0`, `ISOLATION_INCIDENT_SESSION_SCOPE=0`, or `ISOLATION_PROMPTFORGE_SESSION_SCOPE=0` to revert individual resources.
### Pattern 2: Advisory Lock via lockf (P0-3)
macOS ships `lockf(1)` at `/usr/bin/lockf` (not GNU `flock(1)`). Exclusive lock wraps `mc.js ready` invocation; lock released by kernel on process death (SIGKILL-safe per T8 Q1 live test).
**Implementation:** - `mc-ready-gate.sh` (lines 98-112): `lockf -k -t 30 ~/system/state/.ledger-root-hash.lock node ~/system/tools/mc.js ready` - Lock file kept via `-k` flag for reuse - Fail-closed: exits 2 if `lockf` binary absent
**Rollback:** Set `ISOLATION_LEDGER_HASH_FLOCK=0`.
**Why BEGIN IMMEDIATE was required:** - T9 added `PRAGMA busy_timeout` but used DEFERRED transactions (default in sqlite3) - Under w=4 burst, multiple connections acquired SHARED locks simultaneously; first write triggered RESERVED lock race → silent INSERT loss (costs.db) and UPDATE non-determinism (skill-registry.db) - `BEGIN IMMEDIATE` acquires RESERVED lock upfront; only one writer proceeds, others get `SQLITE_BUSY` immediately and retry in application layer
**Expected output post-fix:** - Default mode: P0-1/2/3/5/6 show `LAST_WRITER_WINS` (correct — single fixture path simulates the race), P0-4/7 show `SAFE` - Per-session mode: All 5 per-session P0s (`session_state_ps`, `last_verdict_ps`, `ledger_hash_ps`, `incident_mode_ps`, `prompt_forge_ps`) show `SAFE`
**Verdict location:** `/tmp/session-collision-/probe.jsonl` — each line is a JSON verdict with fields: `ts`, `resource`, `verdict`, `writers`, `pre_hash`, `post_hash`, `lost_writers`, `deadlocked_writers`
### 2. How to Roll Back Any Single Isolation
Set the corresponding feature flag to `0`:
```bash # Roll back P0-1 (SESSION-STATE per-session) export ISOLATION_SESSION_STATE_SCOPED=0
# Roll back P0-4 + P0-7 (SQLite BEGIN IMMEDIATE) export ISOLATION_SQLITE_WAL=0
# Roll back P0-3 (lockf on ledger-root-hash) export ISOLATION_LEDGER_HASH_FLOCK=0 ```
**Validation:** Re-run harness with the flag disabled to confirm rollback worked.
**IMPORTANT:** Rolling back P0-4 or P0-7 restores the LAST_WRITER_WINS collision at w=4. Only roll back if BEGIN IMMEDIATE is causing production deadlocks (none observed in 4 validation runs + 3 stability repeats).
### 3. How to Add a New Shared Resource to Isolation
When a new shared resource is identified (e.g., a new `/tmp/global-marker` file or a new SQLite DB):
**Step 1: Add to inventory**
Edit `~/system/specs/multi-session/shared-state-inventory.md` (T1 artifact): - List the resource path - Classify: `per-session` | `global-single-writer` | `global-multi-writer` | `external-singleton` - Cite the file/line that proves it is touched (e.g., `hook-name.sh:42`)
**Step 2: Write a probe in the harness**
Edit `~/system/tools/diagnose-session-collision.sh`: - Add a `writer_` function that writes to a sandbox fixture - Add a verdict function if the resource needs custom logic (e.g., per-session file enumeration, lock-attempt counting) - Add the resource name to the `TARGETS` array
From `/Users/makinja/system/specs/multi-session/isolation-model.md` §2 (Pattern Catalogue): - **per-session-path:** Single-consumer or append-only state (e.g., session logs) - **advisory-flock (lockf):** Last-writer-wins file with single authoritative value (e.g., a hash file) - **SQLite WAL + BEGIN IMMEDIATE + retry:** SQLite DB with concurrent INSERTs/UPDATEs - **CAS lease (mc.js claim):** Cross-session resource allocation (e.g., task claiming) - **singleton-broker queue:** High-risk writes that need daemon supervision (e.g., MEMORY.md) - **deprecate-and-replace:** The global resource is a design defect; eliminate it
**Step 5: Implement the pattern**
Follow the implementation notes in `isolation-model.md` §4 (Per-P0 Design Table). Add a feature flag (e.g., `ISOLATION_NEW_RESOURCE=1`) for rollback safety.
**Step 6: Validate**
Run `diagnose-session-collision.sh` with the new isolation enabled. Verdict must be `SAFE` at w=4.
**Step 7: Update this runbook**
Add the new resource to the Collision Ledger table above and document the chosen pattern + rollback flag.
---
## Known Limitations
### P1 Resources (13 total) — Not Yet Addressed
From `COLLISION-LEDGER.md` rows 8-17: - `lightrag-ingest-health.json` — SAFE at w=2, LAST_WRITER_WINS at w=4 (2 of 4 increments lost) - `evidence-ledger.jsonl` — not probed; suspected interleaved appends under concurrent `mc.js done` - `evidence-index.jsonl` — not probed; read at session boot without write lock - Mehanik cleared markers (`/tmp/mehanik-cleared-`) — not probed; two sessions on same MC can both see cleared marker - Evidence dirs (`/tmp/evidence-/`) — not probed; numeric sequence collision risk - Claim schema stubs (`/tmp/claim-schema-.json`) — not probed; two sessions on same MC write conflicting schemas - Hop-build started markers (`/tmp/hop-build-started-`) — not probed; 8 stale files present; double-build or skip-build risk - Opus override token (`/tmp/opus-override-token`) — not probed; non-atomic consume allows two sessions to bypass cost gate - John bash override token (`/tmp/john-bash-override-token`) — not probed; same TOCTOU as opus token - MCP Playwright server (singleton) — not probed; unknown whether browser contexts are session-isolated - LightRAG ingest API (`http://localhost:9621`) — not probed; concurrent POST from all sessions; LightRAG's own concurrency handling unverified - MEMORY.md daemon write path — not probed; memory-writer.js queue serialisation under concurrent flush requests
Require Phase 2 sprint 2 or explicit CEO scope expansion.
From `COLLISION-LEDGER.md` rows 18-27: - `blueprint-override-ledger.jsonl`, `h-ready-audit.jsonl`, `verdict-ledger.jsonl`, `daily-logs/.md`, `GOTCHA-task-.md`, `hivemind.db`, `knowledge.db`, `session-save.log` - No CEO-visible blast radius confirmed in T3 - Deferred to backlog
### MCP Singleton Servers — Unprobed
- Playwright browser: unknown whether page state leaks between concurrent `mcp__playwright__navigate` calls - Docker MCP: unknown whether container state is session-isolated - Spreadsheet MCP: unknown whether workbook handles are session-scoped
Require separate external-service isolation plan.
### Harness Measures /tmp Clones, Not Live State
The collision harness writes to `/tmp/session-collision-/fixtures/`, not production paths. Verdicts are correct for concurrency pattern analysis but do not directly measure live production contention. The harness is a structural test, not a load test.
To measure live contention: inspect hook execution logs (`~/system/memory/logs/hook-execution.log`) for `BUSY_TIMEOUT_HIT` (costs.db) or `SKILL_DB_ERROR_FINAL` (skill-registry.db) occurrences during high-concurrency periods.
---
## Out-of-Scope
The following were explicitly excluded from Phase 2:
1. **P1 resources** (13 items listed above) — require separate plan 2. **P2 resources** (14 items listed above) — backlog 3. **External singletons** (MCP servers, LightRAG, Qdrant, Ollama) — require external-service isolation plan 4. **Hook scratch state** not in T3 probe surface: - MEMORY.md direct write path (protected by mmwb daemon redirect) - `settings.local.json` (CEO-only writes per T1 classification) 5. **Legacy /tmp markers** cleanup (8 stale `hop-build-started-*` files present) — cleanup cron needed but collision risk unprobed
No existing hook was removed in Phase 2. Any future removal requires named CEO approval.
---
## Architecture Notes
### Why lockf, Not flock?
macOS 25.2.0 does not ship `flock(1)` (util-linux). macOS provides `lockf(1)` at `/usr/bin/lockf`, which uses BSD `flock(2)` kernel primitive. Semantics: - `flock -x lockfile cmd` → `lockf -k -t 30 lockfile cmd` - `-k` keeps the lock file on exit (required for reuse) - `-t N` sets timeout in seconds (0 = non-blocking) - Lock is released by kernel on any process death (SIGKILL-safe, confirmed by T8 Q1 live test + POSIX spec)
### Why BEGIN IMMEDIATE, Not Just PRAGMA busy_timeout?
SQLite default transaction mode is DEFERRED: `BEGIN DEFERRED` acquires no locks until the first write. Under w=4 burst with WAL mode: 1. Four connections open 2. Each executes `PRAGMA busy_timeout=5000` 3. Each executes `INSERT` (implicit BEGIN DEFERRED) 4. All four acquire SHARED locks 5. First write attempts to upgrade to RESERVED — succeeds 6. Other three attempt upgrade — all get SQLITE_BUSY 7. **But** the PRAGMA busy_timeout retry only applies if the lock was unavailable at BEGIN time. Since all four acquired SHARED before any write, the retry mechanism is bypassed.
Result: 1 of 4 INSERTs succeeds, 3 fail silently (exit code 5 from sqlite3 CLI, which hook may not check).
`BEGIN IMMEDIATE` acquires RESERVED lock upfront. Only one connection gets RESERVED; others block (or get SQLITE_BUSY) at BEGIN, where busy_timeout applies correctly. Application-layer retry loop ensures all writers eventually succeed.
### Why Compaction Only at Boot (P0-1)?
Per-session `SESSION-STATE-.md` files accumulate during the day. Compaction at boot (not at every session end) minimizes file I/O. The 4h mtime staleness filter ensures dead sessions' files are ignored. Compaction uses atomic write (`tmp+mv`) to prevent partial-write corruption if `enforce-next-steps.sh` is killed mid-boot.
### Why 4h Staleness Filter?
Claude Code sessions under normal use are ≤ 2h (median ~30min, p95 ~90min per session log analysis). 4h allows for extended debugging sessions (e.g., CEO deep-dive on a single task) while filtering overnight orphans. Session files older than 4h at boot time are assumed stale and skipped in compaction.
### WAL Sidecar Files
WAL mode creates `-wal` and `-shm` sidecar files next to each SQLite DB: - `-wal`: Write-Ahead Log (contains uncommitted writes) - `-shm`: Shared memory index (used by readers to find data in WAL)
**NEVER manually delete these files while any Claude Code session is running.** Deleting them corrupts the DB. macOS purges `/tmp` on reboot, but `~/system/databases/` is persistent — sidecar files remain until a checkpoint flushes them.
To verify WAL mode is active: ```bash sqlite3 ~/system/databases/costs.db "PRAGMA journal_mode;" # Output: wal ```
To revert to DELETE mode (NOT recommended unless WAL is causing issues): ```bash sqlite3 ~/system/databases/costs.db "PRAGMA journal_mode=DELETE;" ```
---
## Evidence Files
All referenced evidence paths are archived in `~/system/specs/multi-session/`:
- [MC Claim Protocol](https://docs.alai.no/books/infrastructure/page/mc-claim-protocol) — Cross-session task claiming via CAS lease (already production before this work) - [ADR-024 Agent Team Topology](https://docs.alai.no/books/system-architecture/page/agent-team-topology-adr-024) — Agent process supervision (single-session scope) - [ZAKON NULA](https://docs.alai.no/books/rules/page/zakon-nula-tool-first) — Tool-first doctrine that drove the debug-before-solution mandate (T6 phase gate)
---
**Created:** 2026-05-18 **Last Updated:** 2026-05-18 **Plan:** `/Users/makinja/system/specs/claude-code-multi-session-isolation-plan.md` (207 lines) **MC Parent:** #101305 (Phase 2) **Evidence Integrity:** All verdicts cite probe.jsonl line numbers; no LLM inference in ledger or validation