LightRAG Stabilization Runbook — 2026-05-08

Genesis

On 2026-05-08 at 14:00, Kelsey Hightower reported LightRAG returning 502 errors. By 19:05 the service had degraded to complete timeout (000). Root cause: MainThread synchronous list comprehension in lightrag/lightrag.py:872 (apipeline_process_enqueue_documents) iterating over 125,341-record JsonDocStatusStorage on the asyncio event loop. A single POST to /documents/text triggered full file rewrite + pipeline iteration over 121K pending docs → CPU pegged at 100% → /health unreachable. The issue was compounded by running sbnb/lightrag:latest amd64-only image under Rosetta on Apple Silicon, incurring 2-3× performance tax.

Six-Step Fix Applied

S1: Disable Runaway Ingest Agents

Stopped LaunchAgents: com.alai.lightrag-outbox-ingest, com.alai.lightrag-migrate-pump, com.alai.lightrag-watchdog. Kept: keepwarm, backup, monitor.

S2: Prune Pending Queue

Stopped container, backed up doc_status.json. Filtered to status=processed only: 8,357 records retained; 116,986 pending/processing/failed quarantined to backup. Restarted container. CPU dropped to 0.31%.

S3: Verify Queryability

Tested naive mode + only_need_context=true (bypasses LLM, returns ALAI corpus chunks). Graph/label endpoint returned 200+ entities. Service functionally restored.

S4: Image Swap for Native ARM64

Replaced sbnb/lightrag:latest (amd64, v1.3.4) with ghcr.io/hkuds/lightrag:latest (native arm64, v1.4.16, official upstream). Verified via docker manifest inspect.

S5: Resource Limits

Added cgroup-enforced limits in compose: cpus: 2.0, memory: 4G.

S6: Re-Ingest Worker Design

Designed (not implemented) re-ingest worker with: batch_size=10, cooldown=60s, health-gate, pre-flight LLM availability check, cursor-based restart safety. Build gated on CEO OCD-3 (aging policy decision).

Verified Post-State

• Container: Up, healthy
• CPU: 4.29%
• Memory: 1.24/4 GiB
• /health: HTTP 200 in 3.7ms
• Naive query: Returns ALAI documentation chunks
• Knowledge graph: Queryable

Known Follow-Ups

• Child MC #100027 (M, FlowForge): LLM_MODEL=qwen3:8b-q8_0 in .env not on Ollama; hybrid/local/global modes return 404. Fix: one-line .env change to llama3.1:8b OR ollama pull qwen3:8b-q8_0.
• CEO OCD-3 aging policy: Decision required before re-ingest build. 100% of 117K backlog has unknown_source. Option A (drop-all) recommended due to zero provenance utility.
• Martin's asyncio event-loop freeze: HKUDS 1.4.16 may have fixed root cause. Verify before relaxing batch_size in re-ingest worker.

Evidence Files (Local, Transient)

• /tmp/lightrag-stabilization-step1-evidence.txt — launchctl list before/after
• /tmp/lightrag-stabilization-step2-evidence.txt — doc_status counts, /health timing post-restart
• /tmp/lightrag-stabilization-step3-evidence.txt — query mode results, graph endpoint outputs
• /tmp/lightrag-stabilization-step4-evidence.txt — manifest inspect, docker inspect
• /tmp/lightrag-stabilization-step5-evidence.txt — docker stats with limits, compose diff
• /tmp/lightrag-stabilization-step6-evidence.txt — re-ingest worker design doc
• /tmp/lightrag-stabilization-progress.txt — step-by-step progress log
• /tmp/cache-proxy-99981/ — pre-existing Phase 1 hash-cache proxy evidence (separate)

References

• Forge file: /Users/makinja/system/prompts/forged/99982.md (panel decisions used to drive stabilization)
• Mehanik clearance: /tmp/mehanik-cleared-100009 (2026-05-08 19:56)
• Genesis SENTINEL v3 audit: project_sentinel_v3_audit_2026-05-01.md
• Image diff: sbnb/lightrag:latest @ 1.3.4 → ghcr.io/hkuds/lightrag:latest @ 1.4.16