ANVIL Memory Troubleshooting — Mac Studio

Practical runbook for diagnosing and clearing RAM pressure on ANVIL (Makinja's Mac Studio, M3 Ultra, 96GB). Companion to Disk & Memory Health Alarms — What Fires, Where It Lands, How to Test, which documents the CEO-facing Telegram/email alert layer (health-monitor-anvil.js). This page covers the auto-remediation daemons — what actually kills processes and frees RAM, independent of whether anyone gets notified.

Verified against live system state 2026-07-28 08:50 local (tool-first: vm_stat, uptime, plist files, and today's log files were read directly — see Evidence at bottom).

The 4 layers that touch ANVIL memory

Layer	Script	Trigger	What it does
1. Watchdog	`~/system/tools/memory-watchdog.sh`	LaunchAgent `com.alai.memory-watchdog`, every 300s	Free-RAM tiered response: WARN/ALARM/PANIC (see below)
2. Guardian	`~/system/tools/system-guardian.js`	LaunchAgent `com.john.system-guardian`, every 300s	RAM/disk/load/docker auto-optimizer, zombie killer, Ollama idle-unload
3. Ollama guard	`~/system/tools/ollama-memory-guard.js`	LaunchAgent `com.alai.ollama-guard`, every 60s	Enforces max-1-model-loaded in Ollama; unloads all models if RAM ≥ 80%
4. Zombie cleanup	`~/system/tools/zombie-proc-cleanup.sh`	LaunchAgent `com.alai.zombie-cleanup` (hourly) + invoked directly by layer 1's ALARM branch	Kills orphaned `ollama runner` processes and stale `grep -rn` processes (>10 min)

The CEO-alert layer (health-monitor-anvil.js, Telegram → email → log fallback) is a 5th, separate mechanism — see the linked page above. It reads the same vm_stat data but does not perform remediation itself.

Layer 1 — memory-watchdog.sh thresholds (free RAM, of 96GB total)

WARN: free < 15 GB → log + Slack notice only
ALARM: free < 8 GB → runs zombie-proc-cleanup.sh + Slack alarm
PANIC: free < 3 GB → launchctl bootout/bootstrap on com.alai.ollama-serve-v2, SIGTERM on all ollama runner --model and stray grep -rn procs, sync, + Slack page

Layer 2 — system-guardian.js thresholds

RAM: warn 80%, critical 92% (kills zombies + ollama stop at critical)
Disk: warn 75%, critical 85% (Docker image/builder prune at critical; APFS-snapshot-aware)
Load avg (5m): warn 15 (tuned for 24-core M3 Ultra)
Zombie RSS: kills any grep/<defunct> process using > 500MB
Ollama idle-unload: ollama stop after 30 min with no state-file update (wall-clock based, does not depend on the RAM reading)

Layer 3 — ollama-memory-guard.js

Polls GET /api/ps on localhost:11434 every 60s
If Ollama RAM% (measured via its own vm_stat parse) ≥ 80% → unloads all loaded models (keep_alive: 0)
Else if > 1 model loaded → unloads all but the first (oldest survives, rest get unloaded)
This is currently the most reliable of the three remediation daemons (see bugs below) — confirmed live in ~/system/logs/ollama-guard.log actively unloading bge-m3 / llama3.1:8b while keeping qwen2.5-coder:32b.

⚠️ Two known live bugs (verified 2026-07-28, not previously documented)

Both are launchd PATH bugs — the scripts' logic is correct, but the shell environment launchd gives them is missing binaries they call by bare name.

Bug A — memory-watchdog.sh: Slack alerts silently fail

com.alai.memory-watchdog.plist sets no EnvironmentVariables/PATH override, so it runs under launchd's bare default PATH (/usr/bin:/bin:/usr/sbin:/sbin), which does not include /opt/homebrew/bin where node lives on this machine. The script's slack_alert() function shells out to node "$HOME/system/tools/slack.js" ..., which fails with node: command not found on every single WARN/ALARM/PANIC cycle.

Live proof — ~/system/logs/memory-watchdog.log, today (2026-07-28), the 08:29:26 PANIC event:

[2026-07-28 08:29:26] PANIC — free RAM 1.35 GB (< 3 GB threshold)
[2026-07-28 08:29:26] PANIC action 1: brew services restart ollama (kills zombie runners)
[2026-07-28 08:29:28] PANIC action 2: SIGTERM all ollama runner procs + grep -rn procs
[2026-07-28 08:29:28] PANIC action 3: purge page caches (sudo required — may skip)
/Users/makinja/system/tools/memory-watchdog.sh: line 29: node: command not found

The remediation itself (launchctl bootout/bootstrap, pgrep/kill, sync) still runs fine — those are bash builtins or default-PATH binaries. Only the Slack notification is silently dropped, every cycle, at every severity level. This has been happening at least since today's log window and is likely long-standing (the plist has no PATH override at all, so it was never going to work since the tool was written to call node by bare name).

Fix: add an EnvironmentVariables dict to com.alai.memory-watchdog.plist with PATH including /opt/homebrew/bin (mirror what com.john.system-guardian.plist already does), then launchctl kickstart -k gui/501/com.alai.memory-watchdog.

Bug B — system-guardian.js: RAM/load checks always NaN

com.john.system-guardian.plist does set a PATH override, but it is /opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin — it drops /usr/sbin, where sysctl lives. getMemory() and getLoadAvg() in system-guardian.js both shell out to bare sysctl (sysctl -n hw.memsize, sysctl -n vm.loadavg), which fails, so both functions return NaN.

Live proof — ~/system/logs/system-guardian.log, today, every 5-min cycle:

/bin/sh: sysctl: command not found
/bin/sh: sysctl: command not found
[2026-07-28T06:48:24.644Z] [INFO] RAM: NaN/NaNGB (NaN%) | Disk /System/Volumes/Data: 82% (153Gi free) | Load: NaN/NaN/NaN | Docker: 20.2GB

Because NaN >= 92 and NaN >= 15 both evaluate to false in JS, the entire RAM-critical branch (zombie kill + ollama stop) and the load-avg alert never fire, regardless of actual memory pressure. Only the disk-percent branch (uses df, which works) and the wall-clock Ollama-idle-unload timer (doesn't need the RAM reading) still function. In effect, Guardian's headline job — auto-react to real RAM pressure — has been a no-op for as long as this PATH has been set this way.

Fix: add /usr/sbin (and ideally /sbin) to the PATH value in com.john.system-guardian.plist, then launchctl kickstart -k gui/501/com.john.system-guardian.

Net effect of both bugs together

Right now the only layer reliably protecting ANVIL from an Ollama-driven OOM is Layer 3 (ollama-memory-guard.js), because it reads vm_stat itself (in-process, no bare-sysctl/node PATH dependency) rather than shelling out. Layers 1 and 2 are still doing their kill/restart actions on the RAM axis (Layer 1's PANIC branch has no PATH dependency for the actual kills — only its Slack call fails), but Layer 2's RAM branch is fully dead, and Layer 1 is flying blind from a notification standpoint.

Live snapshot at time of writing (2026-07-28, ~08:50 local)

uptime: load averages 12.83 / 11.04 / 9.11 (5m load 11.04 is above Guardian's own WARN=15? No — below 15, but above health-monitor-anvil's cpu_load.warn=8 and alert=12 thresholds — i.e. CPU load has been in ALERT territory on the CEO-facing monitor even though Guardian's own load check is silently broken per Bug B).
memory-watchdog.log shows the system oscillating between OK / WARN / ALARM / PANIC multiple times within a single hour (08:04 ALARM, 08:29 PANIC, 08:49 WARN) — driven by Ollama model loads (qwen2.5-coder:32b + others cycling in ollama-guard.log) competing with Virtualization.framework XPC (Docker Desktop's VM) and Claude Code (claude, node) processes.
This is a recurring, not one-off pattern per the log — the machine is running close to its RAM ceiling under normal multi-agent load.

Manual diagnostic commands

# Current free/used memory (what all the daemons parse)
vm_stat

# Top RSS consumers right now
ps -A -o pid,rss,comm | sort -k2 -rn | head -15

# Ollama's own view of loaded models + RAM
curl -s http://localhost:11434/api/ps | python3 -m json.tool

# Are the daemons actually running?
launchctl list | grep -iE "memory-watchdog|system-guardian|ollama-guard|zombie-cleanup"

# Tail today's remediation activity
tail -50 ~/system/logs/memory-watchdog.log
tail -50 ~/system/logs/system-guardian.log
tail -50 ~/system/logs/ollama-guard.log
tail -30 ~/system/logs/zombie-cleanup.log

# Historical breach data (health-monitor-anvil.js writes here — separate from the two daemons above)
sqlite3 ~/system/databases/health-events.db \
  "SELECT timestamp, check_name, status, value FROM health_events WHERE source='anvil' ORDER BY timestamp DESC LIMIT 20;"

Manual remediation commands (if daemons haven't caught it yet)

# Unload all Ollama models immediately
ollama stop

# Kill a specific stuck ollama runner by PID (from `ps` above)
kill -TERM <pid>

# Restart the Ollama service (equivalent to memory-watchdog.sh's PANIC action 1)
launchctl bootout gui/501/com.alai.ollama-serve-v2 2>/dev/null
launchctl bootstrap gui/501 ~/Library/LaunchAgents/com.alai.ollama-serve-v2.plist

# Kick a specific watchdog daemon to re-run right now
launchctl kickstart -k gui/501/com.alai.memory-watchdog
launchctl kickstart -k gui/501/com.john.system-guardian

Evidence (tool-verified, not from memory)

~/system/tools/memory-watchdog.sh, ~/system/tools/system-guardian.js, ~/system/tools/ollama-memory-guard.js, ~/system/tools/zombie-proc-cleanup.sh, ~/system/tools/health-monitor-anvil.js — read in full 2026-07-28.
~/Library/LaunchAgents/com.alai.memory-watchdog.plist (no PATH override) and ~/Library/LaunchAgents/com.john.system-guardian.plist (PATH missing /usr/sbin) — read in full 2026-07-28.
~/system/logs/memory-watchdog.log, ~/system/logs/system-guardian.log, ~/system/logs/ollama-guard.log, ~/system/logs/zombie-cleanup.log — tailed live 2026-07-28 ~08:50.
vm_stat, uptime — run live 2026-07-28 ~08:50.
BookStack page anvil-memory-troubleshooting-mac-studio (id 2676, book "Operations") — confirmed pre-existing as an empty stub (0 chars markdown, last updated 2026-07-05) before this runbook was written.