Agentic Retrieval — Design skill for Claude Code

Agentic Retrieval Experiment Framework

A framework for measuring how well AI agents retrieve information and reason over large text corpora. Generates controlled experimental corpora with hidden "needles," runs agents against them, scores their responses, and produces analysis-ready data.

Designed for use with [Claude Code](https://docs.anthropic.com/en/docs/claude-code) and the Claude Agent SDK, but the generated corpora and answer keys can be used to evaluate any agent.

Experiment Types

Type	Description
single_needle	Find one hidden fact in the corpus
multi_chain	Follow a chain of N cross-references across files to reach a final value
multi_reasoning	Locate N scattered clues and synthesise them to answer a question

Each type can be parameterized across:

• Content profiles: e.g. Python repository, noir detective fiction
• Corpus sizes: e.g. 20k, 40k, 160k, 800k tokens
• Reference clarity: exact keyword, synonym, contextual paraphrase
• Needle counts: e.g. 2, 8, 16 items (multi-chain and multi-reasoning)
• Discriminability: easy, hard

Define your own parameter grids in `experiments/*.yaml`.

Pipeline

1. Generate — builds realistic background corpora from content profile templates and inserts needle payloads at controlled difficulty levels
2. Run — executes the agent against each corpus with a retrieval question
3. Judge — scores agent responses against answer keys using rubric-based LLM evaluation (correctness + completeness)
4. Analyse — loads verdicts into notebooks for visualisation and interpretation

Getting Started

Prerequisites

• Python 3.12+
• Poetry
• An Anthropic API key (set ANTHROPIC_API_KEY in your environment)

Install

git clone https://github.com/mehrabmodi1/agentic-retrieval.git
cd agentic-retrieval
poetry install

Run Tests

poetry run pytest -v

Genera