iwe-org

Memory Bench — Documentation skill for Claude Code

Documentation community

Benchmarking markdown knowledge graphs as agent memory — LOCOMO and HotPotQA tracks with Claude Code.

How to install Memory Bench

This entry records only its repository, not the path inside it, so there is no exact command to give. Open iwe-org/memory-bench and copy the folder into ~/.claude/skills/, or the file into ~/.claude/agents/.

What Memory Bench does

Benchmarking markdown knowledge graphs as agent memory — LOCOMO and HotPotQA tracks with Claude Code.

Alternatives in Documentation

Notion Knowledge Capture — Convert conversations into structured, searchable Notion wiki entries 14.6k ★
Hyperresearch — Agent-driven research knowledge base 2.4k ★
Apply Self Review — Apply the approved improvements from the latest memory-self-review PROPOSAL 297 ★

README

Markdown knowledge graphs as agent memory

Measures [IWE](https://iwe.md) knowledge graphs — markdown pages joined by links — as memory for `claude -p` (Claude Code headless), across two tracks:

LOCOMO (conversational memory): an agent curates long personal conversations into a store, and answers are produced with that store as the only memory.
HotPotQA (multi-hop retrieval): a document corpus is ingested mechanically, and the measured variable is the graph itself — what link structure is worth, and who has to build it.

Both tracks share the answering architecture: either an agent holding search tools, or (the headline in both) a **single one-shot call over a retrieval dossier** the harness assembles with `iwe retrieve`. Every model interaction — curation, answering, judging — runs through `claude -p`; there are no direct API calls, and sessions authenticate through the bench's own isolated profile. Judge prompts and grading are ported from the legacy [Mem0 LOCOMO evaluation](https://github.com/mem0ai/mem0/tree/aae5989e78/evaluation) (binary CORRECT/WRONG "J" metric).

Headline results (sealed test sets, sonnet-4-6 answering and judging):

track	one-shot over the store	agentic `fs` baseline	verdict
LOCOMO (652 q)	0.7515	0.8113	grep leads by 0.06; one-shot wins efficiency only
HotPotQA (300 q)	0.90–0.91	0.9033	parity at 2.4× lower cost, 5× lower latency

The research narrative for the HotPotQA track is published as [The benchmark that built the tools](https://iwe.md/blog/the-benchmark-that-built-the-tools/); the working draft is `article.md`. Inside this repo: `results/RUNS.md` is the ledger of every measured run and the source of final numbers; `results/curated-pilot-report.md` holds the LOCOMO dev-set analysis; `docs/test-plan.md` (LOCOMO) and `docs/hotpot.md` (HotPotQA) are the protocols of record; `docs/store-form.md` is the normative LOCOMO store shape.