Skip to main content

ANVIL Memory Troubleshooting — Mac Studio

ANVIL Memory Troubleshooting — Mac Studio

Practical runbook for diagnosing and clearing RAM pressure on ANVIL (M2Makinja's UltraMac 192GB)Studio, M3 Ultra, 96GB). Companion to Disk & Memory Health Alarms — What Fires, Where It Lands, How to Test, which documents the CEO-facing Telegram/email alert layer (health-monitor-anvil.js). This page covers the auto-remediation daemons — what actually kills processes and frees RAM, independent of whether anyone gets notified.

Verified against live system state 2026-07-28 08:50 local (tool-first: vm_stat, uptime, plist files, and today's log files were read directly — see Evidence at bottom).

IncidentThe Summary4 layers that touch ANVIL memory

Date:

2026-04-20
Symptom:Systemfreezes,Chrome/Claudeunresponsive,panics
RootCause:Zombie processes+duplicate launchd agents + runaway grep processes
Resolution:tuning,duplicate agent removal, zombie cleanup daemon, Ollama 0.21.0 upgrade

Root Causes

  1. Ollama zombie runners: ollama ps reports 0 models loaded, but pgrep -fl ollama_llama_server shows 4-6 GB processes still resident
  2. Duplicate launchd agents: Both
codebaseshangand
Layer Script Trigger What OOMit kerneldoes
1. Watchdog~/system/tools/memory-watchdog.shLaunchAgent com.alai.memory-watchdog, every 300sFree-RAM tiered response: WARN/ALARM/PANIC (see below)
2. Guardian~/system/tools/system-guardian.jsLaunchAgent com.john.system-guardian, every 300sRAM/disk/load/docker auto-optimizer, zombie killer, Ollama runneridle-unload
3. Ollama configguard ~/system/tools/ollama-memory-guard.js LaunchAgent com.alai.ollama-serve.plistguard, andevery 60sEnforces max-1-model-loaded in Ollama; unloads all models if RAM ≥ 80%
4. Zombie cleanup~/system/tools/zombie-proc-cleanup.shLaunchAgent com.alai.ollama-serve-v2.plistzombie-cleanup running(hourly) simultaneously+ →invoked 2xdirectly Ollamaby daemonslayer
  • grep1's memoryALARM leak:branch
    		• Kills orphaned ollama runner processes and stale grep -rn commandsprocesses on(>10 largemin)
    consume

    The 8+CEO-alert layer (health-monitor-anvil.js, Telegram → email → log fallback) is a 5th, separate mechanism — see the linked page above. It reads the same vm_stat data but does not perform remediation itself.

    Layer 1 — memory-watchdog.sh thresholds (free RAM, of 96GB total)

    • WARN: free < 15 GB RAM→ eachlog + Slack notice only
    • PreloadALARM: warmupfree bloat:< 8 GB → runs zombie-proc-cleanup.sh + Slack alarm
    • PANIC: free < 3 GB → launchctl bootout/bootstrap on com.alai.ollama-serve-v2, SIGTERM on all ollama runner --model and stray grep -rn procs, sync, + Slack page

    Layer 2 — system-guardian.js thresholds

    • RAM: warn 80%, critical 92% (kills zombies + ollama stop at critical)
    • Disk: warn 75%, critical 85% (Docker image/builder prune at critical; APFS-snapshot-aware)
    • Load avg (5m): warn 15 (tuned for 24-core M3 Ultra)
    • Zombie RSS: kills any grep/<defunct> process using > 500MB
    • Ollama idle-unload: ollama stop after 30 min with no state-file update (wall-clock based, does not depend on the RAM reading)

    Layer 3 — ollama-memory-guard.js

    • Polls GET /api/ps on localhost:11434 every 60s
    • If Ollama RAM% (measured via its own vm_stat parse) ≥ 80% → unloads all loaded models (keep_alive: 0)
    • Else if > 1 model loaded → unloads all but the first (oldest survives, rest get unloaded)
    • This is currently the most reliable of the three remediation daemons (see bugs below) — confirmed live in ~/system/logs/ollama-guard.log actively unloading bge-m3 / llama3.1:8b while keeping qwen2.5-coder:32b.

    ⚠️ Two known live bugs (verified 2026-07-28, not previously documented)

    Both are launchd PATH bugs — the scripts' logic is correct, but the shell environment launchd gives them is missing binaries they call by bare name.

    Bug A — memory-watchdog.sh: Slack alerts silently fail

    com.alai.memory-watchdog.plist sets no EnvironmentVariables/PATH override, so it runs under launchd's bare default PATH (/usr/bin:/bin:/usr/sbin:/sbin), which does not include /opt/homebrew/bin where node lives on this machine. The script's slack_alert() function shells out to node "$HOME/system/tools/slack.js" ..., which fails with node: command not found on every single WARN/ALARM/PANIC cycle.

    Live proof — ~/system/logs/memory-watchdog.log, today (2026-07-28), the 08:29:26 PANIC event:

    [2026-07-28 08:29:26] PANIC — free RAM 1.35 GB (< 3 GB threshold)
[2026-07-28 08:29:26] PANIC action 1: brew services restart ollama (kills zombie runners)
[2026-07-28 08:29:28] PANIC action 2: SIGTERM all ollama runner procs + grep -rn procs
[2026-07-28 08:29:28] PANIC action 3: purge page caches (sudo required — may skip)
/Users/makinja/system/tools/memory-watchdog.sh: line 29: node: command not found

    The remediation itself (launchctl bootout/bootstrap, pgrep/kill, sync) still runs fine — those are bash builtins or default-PATH binaries. Only the Slack notification is silently dropped, every cycle, at every severity level. This has been happening at least since today's log window and is likely long-standing (the plist has no PATH override at all, so it was never going to work since the tool was written to call node by bare name).

    Fix: add an EnvironmentVariables dict to com.alai.memory-watchdog.plist with PATH including /opt/homebrew/bin (mirror what com.john.ollama-warmup.system-guardian.plist loadingalready does), then launchctl kickstart -k gui/501/com.alai.memory-watchdog.

    Bug B — system-guardian.js: RAM/load checks always NaN

    com.john.system-guardian.plist does set a PATH override, but it is /opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin — it drops /usr/sbin, where sysctl lives. getMemory() and getLoadAvg() in system-guardian.js both shell out to bare sysctl (sysctl -n hw.memsize, sysctl -n vm.loadavg), which fails, so both functions return NaN.

    Live proof — ~/system/logs/system-guardian.log, today, every 5-min cycle:

    /bin/sh: sysctl: command not found
/bin/sh: sysctl: command not found
[2026-07-28T06:48:24.644Z] [INFO] RAM: NaN/NaNGB (NaN%) | Disk /System/Volumes/Data: 82% (153Gi free) | Load: NaN/NaN/NaN | Docker: 20.2GB

    Because NaN >= 92 and NaN >= 15 both evaluate to false in JS, the entire RAM-critical branch (zombie kill + ollama stop) and the load-avg alert never fire, regardless of actual memory pressure. Only the disk-percent branch (uses df, which works) and the wall-clock Ollama-idle-unload timer (doesn't need the RAM reading) still function. In effect, Guardian's headline job — auto-react to real RAM pressure — has been a no-op for as long as this PATH has been set this way.

    Fix: add /usr/sbin (and ideally /sbin) to the PATH value in com.john.system-guardian.plist, then launchctl kickstart -k gui/501/com.john.system-guardian.

    Net effect of both bugs together

    Right now the only layer reliably protecting ANVIL from an Ollama-driven OOM is Layer 3 models(ollama-memory-guard.js), because it reads vm_stat itself (in-process, no bare-sysctl/node PATH dependency) rather than shelling out. Layers 1 and 2 are still doing their kill/restart actions on bootthe →RAM 48axis GB(Layer baseline1's beforePANIC anybranch workhas no

    PermanentPATH Fixdependency for the actual kills — only its Slack call fails), but Layer 2's RAM branch is fully dead, and Layer 1 is flying blind from a notification standpoint.

    Live snapshot at time of writing (2026-07-28, ~08:50 local)

    • uptime: load averages 12.83 / 11.04 / 9.11 (5m load 11.04 is above Guardian's own WARN=15? No — below 15, but above health-monitor-anvil's cpu_load.warn=8 and alert=12 thresholds — i.e. CPU load has been in ALERT territory on the CEO-facing monitor even though Guardian's own load check is silently broken per Bug B).
    • memory-watchdog.log shows the system oscillating between OK / WARN / ALARM / PANIC multiple times within a single hour (08:04 ALARM, 08:29 PANIC, 08:49 WARN) — driven by Ollama Configmodel loads (qwen2.5-coder:32b + others cycling in ollama-guard.log) competing with Virtualization.framework XPC (Docker Desktop's VM) and Claude Code (claude, node) processes.
    • This is a recurring, not one-off pattern per the log — the machine is running close to its RAM ceiling under normal multi-agent load.

    Manual diagnostic commands

    File:

    # Current free/used memory (what all the daemons parse)
vm_stat

# Top RSS consumers right now
ps -A -o pid,rss,comm | sort -k2 -rn | head -15

# Ollama's own view of loaded models + RAM
curl -s http://localhost:11434/api/ps | python3 -m json.tool

# Are the daemons actually running?
launchctl list | grep -iE "memory-watchdog|system-guardian|ollama-guard|zombie-cleanup"

# Tail today's remediation activity
tail -50 ~/system/logs/memory-watchdog.log
tail -50 ~/system/logs/system-guardian.log
tail -50 ~/system/logs/ollama-guard.log
tail -30 ~/system/logs/zombie-cleanup.log

# Historical breach data (health-monitor-anvil.js writes here — separate from the two daemons above)
sqlite3 ~/system/databases/health-events.db \
  "SELECT timestamp, check_name, status, value FROM health_events WHERE source='anvil' ORDER BY timestamp DESC LIMIT 20;"

    Manual remediation commands (if daemons haven't caught it yet)

    # Unload all Ollama models immediately
ollama stop

# Kill a specific stuck ollama runner by PID (from `ps` above)
kill -TERM <pid>

# Restart the Ollama service (equivalent to memory-watchdog.sh's PANIC action 1)
launchctl bootout gui/501/com.alai.ollama-serve-v2 2>/dev/null
launchctl bootstrap gui/501 ~/Library/LaunchAgents/com.alai.ollama-serve-v2.plist
    


    <?xml# version="1.0"Kick encoding="UTF-8"?>a <!DOCTYPEspecific plistwatchdog PUBLICdaemon "to re-run right now
launchctl kickstart -//Apple//DTDk PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>gui/501/com.alai.ollama-serve-v2</string>memory-watchdog
<key>ProgramArguments</key>launchctl <array>kickstart <string>/usr/local/bin/ollama</string>-k <string>serve</string>gui/501/com.john.system-guardian
</array>
  <key>EnvironmentVariables</key>
  <dict>
    <key>OLLAMA_HOST</key>
    <string>0.0.0.0:11434</string>
    <key>OLLAMA_KEEP_ALIVE</key>
    <string>60s</string>
    <key>OLLAMA_MAX_LOADED_MODELS</key>
    <string>1</string>
    <key>OLLAMA_NUM_PARALLEL</key>
    <string>1</string>
  </dict>
  <key>RunAtLoad</key>
  <true/>
  <key>KeepAlive</key>
  <true/>
  <key>StandardOutPath</key>
  <string>/tmp/ollama-serve.log</string>
  <key>StandardErrorPath</key>
  <string>/tmp/ollama-serve-error.log</string>
</dict>
</plist>
    

    Key
    
parameters:
    Evidence (tool-verified, not from memory)
    

      

    • OLLAMA_KEEP_ALIVE=60s~/system/tools/memory-watchdog.sh — unload model after 60s idle (default 5m causes bloat)
      • 

    • OLLAMA_MAX_LOADED_MODELS=1 — only one model resident at a time
      • 

    • OLLAMA_NUM_PARALLEL=1 — no parallel inference (reduces contention)
      • 

    


    Zombie Cleanup Daemon
    

    File:, ~/Library/LaunchAgents/com.alai.zombie-cleanup.plist
    

    <?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>com.alai.zombie-cleanup</string>
  <key>ProgramArguments</key>
  <array>
    <string>/bin/bash</string>
    <string>/Users/makinja/system/tools/zombie-proc-cleanup.sh<system-guardian.js, ~/string>
  </array>
  <key>StartInterval</key>
  <integer>3600</integer>
  <key>StandardOutPath</key>
  <string>/tmp/zombie-cleanup.log</string>
</dict>
</plist>system/tools/ollama-memory-guard.js
    


    Script:, ~/system/tools/zombie-proc-cleanup.sh
    , 
    #!~/bin/bashsystem/tools/health-monitor-anvil.js # Kill zombie Ollama runners (no parent process or disconnected from ollama serve)
pgrep -fl ollama_llama_server | while— read -rin pidfull rest;2026-07-28.
do
  parent=$(ps -o ppid= -p "$pid" | xargs)
  if [[ -z "$parent" ]] || ! ps -p "$parent" | grep -q ollama; then
    echo "$(date): Killing zombie Ollama runner $pid"
    kill -9 "$pid"
  fi
done

# Kill grep processes older than 5 minutes (likely hung)
ps -eo pid,etime,command | grep 'grep -rn' | while read -r pid etime rest; do
  minutes=$(echo "$etime" | awk -F: '{print ($1*60)+$2}')
  if [[ "$minutes" -gt 5 ]]; then
    echo "$(date): Killing hung grep process $pid (runtime: $etime)"
    kill -9 "$pid"
  fi
done
    


    Disabled Agents
    

  • launchctl unload ~/Library/LaunchAgents/com.alai.ollama-serve.memory-watchdog.plist launchctl(no unloadPATH override) and ~/Library/LaunchAgents/com.john.ollama-warmup.plist
rm ~/Library/LaunchAgents/com.alai.ollama-serve.plist
rm ~/Library/LaunchAgents/com.john.ollama-warmup.system-guardian.plist
    •  
    Ollama(PATH Upgrade
    missing 
    brew/usr/sbin) upgrade— ollama  # 0.19.0 → 0.21.0
# Changelog: Fixed memory leakread in runnerfull cleanup2026-07-28.
(issue #4821)
  • ~/system/logs/memory-watchdog.log
    • 


    OOM Symptom Recognition
    

    Command:
    

    vm_stat | awk '/Pages free/ {printf "%.1f GB\n", $3*16384/1024/1024/1024}'~/system/logs/system-guardian.log
    , 
    Thresholds:
    ~/system/logs/ollama-guard.log, 
      

    • < 5 GB free: Alert~/system/logs/zombie-cleanup.log — investigatetailed toplive memory2026-07-28 consumers~08:50.
      • 

    • <vm_stat, 2 GB free: Criticaluptime — killrun non-essentiallive processes2026-07-28 immediately~08:50.
      • 

    • <BookStack 500page MBanvil-memory-troubleshooting-mac-studio free:(id Imminent2676, OOMbook "Operations") — forceconfirmed quitpre-existing Claude/Chrome,as restartan Ollamaempty stub (0 chars markdown, last updated 2026-07-05) before this runbook was written.
      • 

    

    Quick triage:
    

    ps aux | sort -nrk 4 | head -10  # Top 10 memory hogs
pgrep -fl ollama_llama_server    # Zombie Ollama runners
pgrep -fl grep                    # Hung grep processes
    


    Prevention Checklist
    

      

    1. Monitor free RAM hourly: vm_stat check in cron
      2. 

    2. Zombie cleanup daemon running: launchctl list | grep zombie-cleanup
      3. 

    3. Only one Ollama launchd agent: launchctl list | grep ollama → expect 1 line
      4. 

    4. No warmup preload agents: launchctl list | grep warmup → empty
      5. 

    5. Grep with timeout: timeout 60 grep -rn ... instead of bare grep -rn
      6.
    Back to top