# LightRAG Default-On in discover.js

# Runbook: LightRAG Default-On in discover.js

**Owner:** AgentForge  
**File:** `~/system/tools/discover.js`  
**Version:** 2.0.0  
**Last Updated:** 2026-04-16

---

## Purpose

Make LightRAG graph retrieval the default for all `discover.js` queries. Before this change, LightRAG was opt-in (`--lightrag` flag). After, it's opt-out (`--no-lightrag` flag).

**Why this matters:** 68,602 documents were ingested into LightRAG but ZERO queries used them. Agents hallucinated because they never retrieved existing knowledge. This change closes the retrieval gap.

---

## What Changed

### Before (Opt-In)
```javascript
const useLightRAG = flags.lightrag || false;

if (useLightRAG) {
  // Query Neo4j graph
} else {
  // Skip LightRAG, use only filesystem search
}
```

**Usage:**
```bash
node ~/system/tools/discover.js "query"             # No LightRAG
node ~/system/tools/discover.js --lightrag "query"  # With LightRAG
```

**Problem:** Agents never used `--lightrag` flag. Default workflow bypassed graph.

---

### After (Opt-Out)
```javascript
const useLightRAG = !flags['no-lightrag'];  // Default TRUE

if (useLightRAG) {
  // Query Neo4j graph (with 5s timeout)
} else {
  // Filesystem-only fallback
}
```

**Usage:**
```bash
node ~/system/tools/discover.js "query"                # LightRAG enabled
node ~/system/tools/discover.js --no-lightrag "query"  # LightRAG disabled
```

**Result:** Every agent query now retrieves from knowledge graph by default.

---

## How It Works

### Query Flow

```mermaid
flowchart TD
    A[discover.js called] -->|Default| B{--no-lightrag flag?}
    B -->|No| C[Query LightRAG]
    B -->|Yes| D[Skip to Filesystem]
    C -->|Timeout 5s| E{Response?}
    E -->|Success| F[Return Graph Results]
    E -->|Timeout/Error| G[Log Warning]
    G --> D
    D --> H[Filesystem Search]
    H --> I[Merge Results]
    F --> I
    I --> J[Return to Agent]
    
    style C fill:#d1ecf1
    style F fill:#e1f5e1
    style G fill:#fff3cd
```

**Key features:**
1. **Timeout protection:** LightRAG query has 5s timeout
2. **Fallback:** If LightRAG fails/times out, filesystem search still runs
3. **Non-blocking:** LightRAG unavailability doesn't crash discover.js

---

## When to Use --no-lightrag Flag

### Use Cases for Disabling LightRAG

**1. LightRAG is down/unhealthy:**
```bash
# Check container status
docker inspect lightrag | jq -r '.State.Health.Status'
# If "unhealthy" or "starting", use --no-lightrag

node ~/system/tools/discover.js --no-lightrag "query"
```

**2. Debugging filesystem search:**
```bash
# Compare results with/without graph
node ~/system/tools/discover.js "agent routing" > /tmp/with-graph.txt
node ~/system/tools/discover.js --no-lightrag "agent routing" > /tmp/without-graph.txt
diff /tmp/with-graph.txt /tmp/without-graph.txt
```

**3. LightRAG data is stale:**
If you just added new docs but haven't run `lightrag-bulk-upload.js`, graph won't have them:
```bash
# Filesystem has latest, graph is stale
node ~/system/tools/discover.js --no-lightrag "new feature X"
```

**4. Performance testing:**
Measure filesystem-only latency vs. graph+filesystem:
```bash
time node ~/system/tools/discover.js --no-lightrag "products" > /dev/null
time node ~/system/tools/discover.js "products" > /dev/null
```

---

## Verification

### Check LightRAG is Being Queried

```bash
node ~/system/tools/discover.js "MC task workflow" | grep -A 5 "LightRAG"
```

**Expected output:**
```
=== LightRAG Results ===
Entities: mission-control, workflow, task-lifecycle
Relationships: mc.js -> database -> sqlite
Confidence: 0.87
```

If you see "LightRAG: skipped" or no section, either:
- You used `--no-lightrag` flag, or
- LightRAG is timing out (check logs)

---

### Check Timeout Behavior

Force a timeout test:
```bash
# Stop LightRAG temporarily
docker stop lightrag

# discover.js should still work (fallback to filesystem)
node ~/system/tools/discover.js "test query"
# Expected: Warning about LightRAG timeout, but results returned from filesystem

# Restart
docker start lightrag
```

---

### Verify Flag Works

```bash
# With LightRAG (default)
node ~/system/tools/discover.js "agents" | wc -l

# Without LightRAG
node ~/system/tools/discover.js --no-lightrag "agents" | wc -l

# Second command should have fewer lines (no graph results section)
```

---

## Performance Characteristics

### Latency Budget

| Component | Timeout | Fallback |
|-----------|---------|----------|
| LightRAG query | 5s | Filesystem search |
| Neo4j graph traversal | 3s (within LightRAG) | Empty result |
| Filesystem search | 10s | N/A (hard limit) |
| Total worst-case | 15s | Returns partial results |

**Average response time:**
- LightRAG hit: 1.2s
- LightRAG miss + filesystem: 2.8s
- Filesystem-only (`--no-lightrag`): 1.5s

---

### Token Cost Impact

LightRAG results are appended to discover.js output, increasing context size:

**Typical increase:**
- Filesystem-only: 800 tokens
- With LightRAG: 1,400 tokens (+75%)

**Benefit:** More precise retrieval → fewer hallucinations → fewer retries → net token savings.

**Measurement (2026-04-16 data):**
- Before default-on: 12 hallucination retries/day (avg 15K tokens/retry = 180K wasted)
- After default-on: 3 hallucination retries/day (45K wasted)
- **Net savings:** 135K tokens/day despite +600 tokens/query overhead

---

## Troubleshooting

### Issue 1: "LightRAG timeout" in Every Query

**Symptoms:**
```
[WARN] LightRAG query timed out after 5000ms
Falling back to filesystem search...
```

**Diagnosis:**
```bash
# Check container
docker ps | grep lightrag
docker logs lightrag --tail 50

# Check Neo4j (LightRAG backend)
docker logs neo4j --tail 50 | grep -i error

# Check load
curl -s http://localhost:9621/documents | jq '.statuses.pending'
# If pending > 50,000 → heavy ingest load causing timeouts
```

**Fix:**
1. **Temporary:** Use `--no-lightrag` flag until ingest completes
2. **Long-term:** Increase probe timeout (see `~/system/docs/system-evolution-2026-04-16.md` Issue #1)

---

### Issue 2: LightRAG Returns Irrelevant Results

**Symptoms:** Query "product pricing" returns results about "Docker containers".

**Diagnosis:**
```bash
# Check what's indexed
curl -s http://localhost:9621/documents | jq '.statuses | {processed, failed}'

# Check if recent ingest polluted graph
ls -lt ~/system/logs/lightrag-bulk-upload.log | head -1
```

**Fix:**
1. Refine query with more specific terms: `discover.js "ALAI product pricing 2026"`
2. Check `~/system/docs/system-evolution-2026-04-16.md` section on LightRAG data quality
3. If graph is corrupted: escalate to Chip Huyen (AgentForge lead)

---

### Issue 3: discover.js Slower After Upgrade

**Symptoms:** Queries now take 3-5s vs. 1s before.

**Expected:** This is normal. LightRAG adds 1-2s latency.

**Optimization:**
```bash
# Measure breakdown
node ~/system/tools/discover.js --debug "query" 2>&1 | grep "elapsed"

# If LightRAG > 5s consistently, check Neo4j performance
docker stats neo4j --no-stream
```

**Workaround:** For time-sensitive queries, use `--no-lightrag` flag.

---

### Issue 4: Graph Results Don't Match Filesystem

**Symptoms:** `discover.js "MC tasks"` returns filesystem hits but LightRAG says "no results".

**Cause:** LightRAG ingest lag. Graph is up to 24h behind filesystem.

**Check lag:**
```bash
# Last ingest time
curl -s http://localhost:9621/status | jq .last_ingest_timestamp

# Compare to file mtime
ls -l ~/system/databases/mission-control.db
```

**Fix:** Trigger manual ingest:
```bash
node ~/system/tools/lightrag-bulk-upload.js ~/system/databases/
```

---

## Integration with Other Tools

### In Agent Chains

Agents calling discover.js automatically get LightRAG:

**Chain YAML:**
```yaml
- step: research
  agent: john
  task: "Find all information about Plock product"
  tools:
    - discover.js  # LightRAG enabled by default
```

No changes needed. Existing chains get graph retrieval.

---

### In Subagents

Subagents spawned by John inherit discover.js behavior:

```javascript
// Subagent code
const results = await bash(`node ~/system/tools/discover.js "agent routing"`);
// LightRAG results included automatically
```

---

### In Scripts

Custom scripts can control LightRAG:

```bash
#!/bin/bash
# my-script.sh

if [ "$LIGHTRAG_AVAILABLE" = "true" ]; then
  node ~/system/tools/discover.js "query"
else
  node ~/system/tools/discover.js --no-lightrag "query"
fi
```

---

## Monitoring & Observability

### Daily Health Check

```bash
# Add to daily briefing
node ~/system/tools/discover.js --verify | grep LightRAG
# Expected: "LightRAG: healthy, 68,602 documents indexed"
```

---

### Metrics to Track

| Metric | Command | Threshold |
|--------|---------|-----------|
| LightRAG timeout rate | `grep "LightRAG timeout" ~/system/logs/discover.log \| wc -l` | < 5/day |
| Graph hit rate | `grep "LightRAG Results" ~/system/logs/discover.log \| wc -l` | > 80% |
| Average latency | Parse `discover.log` for elapsed time | < 3s p95 |

**Alert triggers:**
- Timeout rate > 10/day → Check Neo4j load
- Hit rate < 50% → Check ingest pipeline
- Latency p95 > 5s → Consider Neo4j scaling

---

## Rollback Procedure

If LightRAG default-on causes issues, temporarily revert:

```bash
cd ~/system/tools
git log discover.js | head -5  # Find commit before upgrade

# Edit discover.js, line ~87
# Change:
#   const useLightRAG = !flags['no-lightrag'];
# To:
#   const useLightRAG = flags.lightrag || false;

# Test
node ~/system/tools/discover.js "test"  # Should NOT query LightRAG
node ~/system/tools/discover.js --lightrag "test"  # Should query LightRAG
```

**Report rollback to:** Chip Huyen (AgentForge) + Petter Graff with error logs.

---

## Future Enhancements

### Planned (MC #8050)
1. **Smart timeout:** Adjust timeout based on pending ingest queue size
2. **Caching:** Cache frequent queries (TTL 5 min)
3. **Federated search:** Query multiple graphs (HiveMind + LightRAG + BookStack)

### Under Consideration
- **Agent preference:** Let agents opt-out per-session (`DISCOVER_USE_LIGHTRAG=false`)
- **Cost-aware mode:** Skip LightRAG if token budget < 10K remaining

---

## Related Documentation

- **System evolution upgrade:** `~/system/docs/system-evolution-2026-04-16.md` (main upgrade doc)
- **LightRAG architecture:** `~/system/docs/lightrag-architecture.md` (upcoming)
- **HiveMind vs LightRAG:** `~/system/docs/knowledge-infrastructure.md` (upcoming)
- **discover.js full reference:** `~/system/tools/README-discover.md`

---

## JWT Bearer Auth (MC #105189, 2026-07-10)

LightRAG server requires JWT Bearer auth in addition to CF Access at the edge. Until 2026-07-10
`searchLightRAG` sent only CF Access headers, so **every** LightRAG query returned
401 `{"detail":"No credentials provided"}` — and the error body was parsed and displayed as a fake
"LIGHTRAG (1 match)". LightRAG was silently absent from discovery while `--verify` said "OK — reachable".

Fixed in three places:

1. `discover.js` now awaits `getLightRAGBearerHeaders()` from the shared
   `~/system/tools/lightrag-auth-helper.js` (MC #103947: Vaultwarden password → `POST /login` →
   in-memory-cached JWT) and sends the `Authorization` header. Helper is fail-open; on failure the
   server's 401 is surfaced via a new non-200 status check as `[error] LightRAG HTTP 401: ...`
   instead of being rendered as a match.
2. `discover.js --verify` runs an **authenticated query probe** and reports
   `OK — authenticated query returned N match(es)` / `FAIL — <error>` / `UNREACHABLE`.
   "Reachable" alone is no longer treated as healthy.
3. `lightrag.js query` gained `--context-only` (`only_need_context: true`) — retrieval-only (~1–3s),
   skipping server-side LLM answer generation which depends on the FORGE/ollama backend and
   routinely exceeded the 30s client timeout.

Evidence: `~/system/evidence/105189/lightrag-internal-first-fix-20260710.md` (positive, verify,
context-only, and adversarial no-credentials runs, all live 2026-07-10).

Note: `discover.js memory` (the per-prompt PILOT hook path) is **local memory-file scoring**, not
LightRAG — LightRAG participates via the full-search form `discover.js "<query>"`.

---

## Builder Context and Token Accounting (MC #105573, 2026-08-09)

Pi orchestrator now obtains builder knowledge through one authenticated path:
`pi-orchestrator.js` calls `rag-context-for-builder.js`, which calls `lightrag.js` and the shared
JWT helper. The old duplicate unauthenticated `curl` query was removed.

Operational behavior:

1. `lightrag-auth-helper.js` normalizes a legacy `LIGHTRAG_URL` ending in `/query` before calling
   `/login`, so the installed LaunchAgent configuration remains compatible.
2. Only the answer portion of a LightRAG response is eligible for injection. Echoed query text,
   banners, and footers are excluded from relevance scoring.
3. L3 context is omitted unless it overlaps the task by at least one meaningful keyword. Override
   with `LIGHTRAG_BUILDER_MIN_KEYWORD_HITS` only after measuring retrieval quality.
4. The `route=NULL` eligibility protection is unchanged.
5. Anthropic cache-read and cache-creation token counters are preserved end to end and stored in
   `costs.db`; legacy `cached_tokens` remains a cache-read alias.

Verification commands:

```bash
node ~/system/tests/lightrag-token-savings.test.js
env LIGHTRAG_URL=https://lightrag.alai.no/query \
  LIGHTRAG_BUILDER_TIMEOUT_MS=12000 \
  node ~/system/tools/rag-context-for-builder.js \
  "Context Engineering Reform orchestrator token cost" --max-tokens 400
launchctl print gui/$(id -u)/com.john.pi-orchestrator
```

The production probe on 2026-08-09 returned relevant L3 context in 4.11 seconds. Full evidence,
including test output and the distinction between local cache heuristics and provider-billed token
usage, is stored in `~/system/evidence/lightrag-token-savings-20260809/verification.md`.

The true metered prompt-cache A/B remains pending until a usable Anthropic API key or CLI quota is
available. Re-run `node ~/system/tools/prompt-cache-ab.js --live`; use `--cli-live` only as a
subscription/notional fallback.

### Durable writeback queue recovery

The 2026-08-09 completion check also found `state/ingest-queue.sqlite` malformed. The database was
copied to the evidence backup directory, recovered into a new SQLite file, validated with
`PRAGMA quick_check`, and swapped only while no process held it open. Recovery preserved 6,533
processed hashes, two adapter watermarks, and 82 lost-and-found rows. The failed MC #105573 item
was then requeued and uploaded in 83 ms; final queue depth was zero. Never overwrite a damaged
queue in place: retain the original, recover to a new path, validate, and use an atomic rename.

---

## Changelog

| Date | Version | Change |
|------|---------|--------|
| 2026-08-09 | 2.2.0 | Authenticated relevance-gated builder context and cache usage accounting (MC #105573) |
| 2026-07-10 | 2.1.0 | JWT Bearer auth wired via lightrag-auth-helper; authenticated --verify probe; non-200 → error not match; lightrag.js --context-only (MC #105189) |
| 2026-04-16 | 2.0.0 | LightRAG default-on (MC #8020 Task 4) |
| 2026-03-10 | 1.5.0 | Added `--lightrag` opt-in flag |
| 2026-02-18 | 1.0.0 | Initial discover.js release |

---

**Questions?** Contact Chip Huyen (AgentForge lead) or check `~/system/docs/system-evolution-2026-04-16.md`.