ANVIL Memory Troubleshooting — Mac Studio

ANVIL Memory Troubleshooting — Mac Studio 
 Practical runbook for diagnosing and clearing RAM pressure on ANVIL (Makinja's Mac Studio, M3 Ultra, 96GB). Companion to Disk & Memory Health Alarms — What Fires, Where It Lands, How to Test , which documents the CEO-facing Telegram/email alert layer ( health-monitor-anvil.js ). This page covers the auto-remediation daemons — what actually kills processes and frees RAM, independent of whether anyone gets notified. 
 Verified against live system state 2026-07-28 08:50 local (tool-first: vm_stat , uptime , plist files, and today's log files were read directly — see Evidence at bottom). 
 
 The 4 layers that touch ANVIL memory 
 
 
 
 Layer 
 Script 
 Trigger 
 What it does 
 
 
 
 
 1. Watchdog 
 ~/system/tools/memory-watchdog.sh 
 LaunchAgent com.alai.memory-watchdog , every 300s 
 Free-RAM tiered response: WARN/ALARM/PANIC (see below) 
 
 
 2. Guardian 
 ~/system/tools/system-guardian.js 
 LaunchAgent com.john.system-guardian , every 300s 
 RAM/disk/load/docker auto-optimizer, zombie killer, Ollama idle-unload 
 
 
 3. Ollama guard 
 ~/system/tools/ollama-memory-guard.js 
 LaunchAgent com.alai.ollama-guard , every 60s 
 Enforces max-1-model-loaded in Ollama; unloads all models if RAM ≥ 80% 
 
 
 4. Zombie cleanup 
 ~/system/tools/zombie-proc-cleanup.sh 
 LaunchAgent com.alai.zombie-cleanup (hourly) + invoked directly by layer 1's ALARM branch 
 Kills orphaned ollama runner processes and stale grep -rn processes (>10 min) 
 
 
 
 The CEO-alert layer ( health-monitor-anvil.js , Telegram → email → log fallback) is a 5th, separate mechanism — see the linked page above. It reads the same vm_stat data but does not perform remediation itself. 
 Layer 1 — memory-watchdog.sh thresholds (free RAM, of 96GB total) 
 
 WARN : free < 15 GB → log + Slack notice only 
 ALARM : free < 8 GB → runs zombie-proc-cleanup.sh + Slack alarm 
 PANIC : free < 3 GB → launchctl bootout / bootstrap on com.alai.ollama-serve-v2 , SIGTERM on all ollama runner --model and stray grep -rn procs, sync , + Slack page 
 
 Layer 2 — system-guardian.js thresholds 
 
 RAM: warn 80%, critical 92% (kills zombies + ollama stop at critical) 
 Disk: warn 75%, critical 85% (Docker image/builder prune at critical; APFS-snapshot-aware) 
 Load avg (5m): warn 15 (tuned for 24-core M3 Ultra) 
 Zombie RSS: kills any grep / <defunct> process using > 500MB 
 Ollama idle-unload: ollama stop after 30 min with no state-file update (wall-clock based, does not depend on the RAM reading) 
 
 Layer 3 — ollama-memory-guard.js 
 
 Polls GET /api/ps on localhost:11434 every 60s 
 If Ollama RAM% (measured via its own vm_stat parse) ≥ 80% → unloads all loaded models ( keep_alive: 0 ) 
 Else if > 1 model loaded → unloads all but the first (oldest survives, rest get unloaded) 
 This is currently the most reliable of the three remediation daemons (see bugs below) — confirmed live in ~/system/logs/ollama-guard.log actively unloading bge-m3 / llama3.1:8b while keeping qwen2.5-coder:32b . 
 
 
 ⚠️ Two known live bugs (verified 2026-07-28, not previously documented) 
 Both are launchd PATH bugs — the scripts' logic is correct, but the shell environment launchd gives them is missing binaries they call by bare name. 
 Bug A — memory-watchdog.sh: Slack alerts silently fail 
 com.alai.memory-watchdog.plist sets no EnvironmentVariables /PATH override , so it runs under launchd's bare default PATH ( /usr/bin:/bin:/usr/sbin:/sbin ), which does not include /opt/homebrew/bin where node lives on this machine. The script's slack_alert() function shells out to node "$HOME/system/tools/slack.js" ... , which fails with node: command not found on every single WARN/ALARM/PANIC cycle . 
 Live proof — ~/system/logs/memory-watchdog.log , today (2026-07-28), the 08:29:26 PANIC event: 
 [2026-07-28 08:29:26] PANIC — free RAM 1.35 GB (< 3 GB threshold)
[2026-07-28 08:29:26] PANIC action 1: brew services restart ollama (kills zombie runners)
[2026-07-28 08:29:28] PANIC action 2: SIGTERM all ollama runner procs + grep -rn procs
[2026-07-28 08:29:28] PANIC action 3: purge page caches (sudo required — may skip)
/Users/makinja/system/tools/memory-watchdog.sh: line 29: node: command not found
 
 The remediation itself ( launchctl bootout / bootstrap , pgrep / kill , sync ) still runs fine — those are bash builtins or default-PATH binaries. Only the Slack notification is silently dropped , every cycle, at every severity level. This has been happening at least since today's log window and is likely long-standing (the plist has no PATH override at all, so it was never going to work since the tool was written to call node by bare name). 
 Fix: add an EnvironmentVariables dict to com.alai.memory-watchdog.plist with PATH including /opt/homebrew/bin (mirror what com.john.system-guardian.plist already does), then launchctl kickstart -k gui/501/com.alai.memory-watchdog . 
 Bug B — system-guardian.js: RAM/load checks always NaN 
 com.john.system-guardian.plist does set a PATH override, but it is /opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin — it drops /usr/sbin , where sysctl lives. getMemory() and getLoadAvg() in system-guardian.js both shell out to bare sysctl ( sysctl -n hw.memsize , sysctl -n vm.loadavg ), which fails, so both functions return NaN . 
 Live proof — ~/system/logs/system-guardian.log , today, every 5-min cycle: 
 /bin/sh: sysctl: command not found
/bin/sh: sysctl: command not found
[2026-07-28T06:48:24.644Z] [INFO] RAM: NaN/NaNGB (NaN%) | Disk /System/Volumes/Data: 82% (153Gi free) | Load: NaN/NaN/NaN | Docker: 20.2GB
 
 Because NaN >= 92 and NaN >= 15 both evaluate to false in JS, the entire RAM-critical branch (zombie kill + ollama stop ) and the load-avg alert never fire , regardless of actual memory pressure. Only the disk-percent branch (uses df , which works) and the wall-clock Ollama-idle-unload timer (doesn't need the RAM reading) still function. In effect, Guardian's headline job — auto-react to real RAM pressure — has been a no-op for as long as this PATH has been set this way. 
 Fix: add /usr/sbin (and ideally /sbin ) to the PATH value in com.john.system-guardian.plist , then launchctl kickstart -k gui/501/com.john.system-guardian . 
 Net effect of both bugs together 
 Right now the only layer reliably protecting ANVIL from an Ollama-driven OOM is Layer 3 ( ollama-memory-guard.js ), because it reads vm_stat itself (in-process, no bare- sysctl / node PATH dependency) rather than shelling out. Layers 1 and 2 are still doing their kill/restart actions on the RAM axis (Layer 1's PANIC branch has no PATH dependency for the actual kills — only its Slack call fails), but Layer 2's RAM branch is fully dead, and Layer 1 is flying blind from a notification standpoint. 
 
 Live snapshot at time of writing (2026-07-28, ~08:50 local) 
 
 uptime : load averages 12.83 / 11.04 / 9.11 (5m load 11.04 is above Guardian's own WARN=15? No — below 15, but above health-monitor-anvil's cpu_load.warn=8 and alert=12 thresholds — i.e. CPU load has been in ALERT territory on the CEO-facing monitor even though Guardian's own load check is silently broken per Bug B). 
 memory-watchdog.log shows the system oscillating between OK / WARN / ALARM / PANIC multiple times within a single hour (08:04 ALARM, 08:29 PANIC, 08:49 WARN) — driven by Ollama model loads ( qwen2.5-coder:32b + others cycling in ollama-guard.log ) competing with Virtualization.framework XPC (Docker Desktop's VM) and Claude Code ( claude , node ) processes. 
 This is a recurring, not one-off pattern per the log — the machine is running close to its RAM ceiling under normal multi-agent load. 
 
 
 Manual diagnostic commands 
 # Current free/used memory (what all the daemons parse)
vm_stat

# Top RSS consumers right now
ps -A -o pid,rss,comm | sort -k2 -rn | head -15

# Ollama's own view of loaded models + RAM
curl -s http://localhost:11434/api/ps | python3 -m json.tool

# Are the daemons actually running?
launchctl list | grep -iE "memory-watchdog|system-guardian|ollama-guard|zombie-cleanup"

# Tail today's remediation activity
tail -50 ~/system/logs/memory-watchdog.log
tail -50 ~/system/logs/system-guardian.log
tail -50 ~/system/logs/ollama-guard.log
tail -30 ~/system/logs/zombie-cleanup.log

# Historical breach data (health-monitor-anvil.js writes here — separate from the two daemons above)
sqlite3 ~/system/databases/health-events.db \
 "SELECT timestamp, check_name, status, value FROM health_events WHERE source='anvil' ORDER BY timestamp DESC LIMIT 20;"
 
 Manual remediation commands (if daemons haven't caught it yet) 
 # Unload all Ollama models immediately
ollama stop

# Kill a specific stuck ollama runner by PID (from `ps` above)
kill -TERM <pid>

# Restart the Ollama service (equivalent to memory-watchdog.sh's PANIC action 1)
launchctl bootout gui/501/com.alai.ollama-serve-v2 2>/dev/null
launchctl bootstrap gui/501 ~/Library/LaunchAgents/com.alai.ollama-serve-v2.plist

# Kick a specific watchdog daemon to re-run right now
launchctl kickstart -k gui/501/com.alai.memory-watchdog
launchctl kickstart -k gui/501/com.john.system-guardian
 
 
 Evidence (tool-verified, not from memory) 
 
 ~/system/tools/memory-watchdog.sh , ~/system/tools/system-guardian.js , ~/system/tools/ollama-memory-guard.js , ~/system/tools/zombie-proc-cleanup.sh , ~/system/tools/health-monitor-anvil.js — read in full 2026-07-28. 
 ~/Library/LaunchAgents/com.alai.memory-watchdog.plist (no PATH override) and ~/Library/LaunchAgents/com.john.system-guardian.plist (PATH missing /usr/sbin ) — read in full 2026-07-28. 
 ~/system/logs/memory-watchdog.log , ~/system/logs/system-guardian.log , ~/system/logs/ollama-guard.log , ~/system/logs/zombie-cleanup.log — tailed live 2026-07-28 ~08:50. 
 vm_stat , uptime — run live 2026-07-28 ~08:50. 
 BookStack page anvil-memory-troubleshooting-mac-studio (id 2676, book "Operations") — confirmed pre-existing as an empty stub (0 chars markdown, last updated 2026-07-05) before this runbook was written.