Skip to main content

discover.js Re-Index + LightRAG Fallback

discover.js — Re-Index + LightRAG Fallback

Owner: AgentForge Implemented: 2026-04-17 (Hive Activation Phase 4 T9 + T10) Tool: ~/system/tools/discover.js Index: ~/system/tools/.alai/discover-index.json Post-sync wrapper: ~/system/tools/library-sync-wrapper.sh

Purpose

Before T9, discover.js "drop" returned 3 product hits and 0 hits across tools/skills/agents/MCP/BookStack. Index was slow and shallow.

After T9+T10: persistent inverted index (521 entries from 6 sources), sub-50ms queries, and a semantic LightRAG fallback when the local index is thin.

Sources indexed

Source Count (first build) Origin file
tools 206 ~/system/tools/manifest-index.md + manifest.md
skills 64 ~/system/databases/skill-registry.db
agents 22 ~/system/agents/specialist-mapping.json + ~/.claude/agents/*.md
mcp 7 ~/.claude.json .mcpServers
bookstack 182 ~/system/config/bookstack-sync-map.json
products 40 ~/system/data/product-index.json

Rebuild

# Manual
node ~/system/tools/discover.js --rebuild-index

# Automatic — happens every 5 min via library-sync-wrapper.sh,
# which is what com.alai.library-sync plist invokes after library sync

Atomic writes: index is written to .tmp, then renamed. No partial state visible to readers.

Query behavior

node ~/system/tools/discover.js "<query>"
# → tokens match inverted index → grouped by source

node ~/system/tools/discover.js "<query>" --no-lightrag
# → suppress LightRAG fallback entirely

If total hits across (tools + skills + agents + mcp + bookstack) < 3, the script queries LightRAG with a 5-second timeout. Results are prefixed LIGHTRAG (fallback — semantic) so you can tell them apart from keyword matches.

If LightRAG is slow or unavailable, the fallback silently times out and returns whatever the local index had. No hang.

Known issues

  • First rebuild takes ~2s on current corpus. If total entries grow > 10k, consider a stemmed index or move to SQLite FTS5.
  • products source expects ~/system/data/product-index.json; if missing, that category prints 0 hits. Fine — not a crash.
  • LightRAG fallback depends on the drain from System Evolution T6 finishing (65k pending). While the drain is running, most semantic queries will time out and silently return nothing — that's the designed behavior.
  • System Evolution T4 made LightRAG default-on. T10 here adds the conditional fallback.
  • Library auto-push (runbook library-auto-push.md) runs the wrapper that rebuilds this index.

No comments to display

Back to top