Part 21: Real-Time Knowledge Sync — Event-Driven LightRAG Updates

Part 21: Real-Time Knowledge Sync — Event-Driven LightRAG Updates

*No cron. No polling. File changes hit your knowledge graph in under 6 seconds.*

---

The Problem With Batch Ingestion

Part 18 showed you how to set up LightRAG and bulk-ingest your vault. But what happens after that? Every new vault file, every session memory update, every autoDream consolidation — none of it reaches your knowledge graph until you manually re-run the ingestion script.

Most guides tell you to set up a cron job that polls every 5-30 minutes. That's lazy engineering:

undefined

The Fix: Event-Driven File Watcher

Instead of asking "did anything change?" every N minutes, we tell the OS: "notify me the instant something changes." Zero CPU when idle. Reacts in milliseconds.

flowchart LR
    F[File Written/Modified] -->|OS notification| W[File Watcher]
    W -->|3s debounce| D{Changed?}
    D -->|No, same hash| S[Skip]
    D -->|Yes, new content| U{LightRAG up?}
    U -->|Yes| API[Upload via REST API]
    U -->|No| Q[Queue to disk]
    Q -->|LightRAG recovers| API
    API --> KG[(Knowledge Graph Updated)]

Features Built In

Feature	Why It Matters
Event-driven	OS filesystem notifications via watchdog. Zero CPU when idle.
Debouncing	Waits 3 seconds after the last change before uploading. Handles editors that rapid-save.
Deduplication	MD5 hash tracking. If content didn't actually change, skip the upload.
Retry with backoff	Failed uploads retry 3× with exponential backoff (2s, 4s, 8s).
Self-healing	Health monitor checks LightRAG every 60s. Auto-detects outages and recoveries.
Offline queue	When LightRAG is down, changes queue to disk. When it comes back, auto-flushes. Survives restarts.
Size guards	Skips files >100KB (noisy for graph extraction) or <50 chars (empty).
Graceful shutdown	Saves state + queue on SIGINT/SIGTERM. Nothing lost on restart.
Rotating logs	5MB max, 3 backups. Won't fill your disk.

What This Means For You

Write a vault file → it's in your knowledge graph in ~6 seconds. No manual steps. No cron. No "remember to re-index."

Your autoDream writes a new consolidation? In the graph. Your session-memory hook saves today's work? In the graph. You manually create a decision file? In the graph.

---

Setup

Prerequisites

undefined

Install Dependencies

pip install watchdog requests

Config