mehrabmodi1

Agentic Retrieval

Design community

SOURCE https://github.com/mehrabmodi1/agentic-retrieval

Description

Experiment suite to analyse agentic retrieval and reasoning from text corpora. Allows flexible experiment design, but comes with 3 experiment types of increasing retrieval/reasoning difficulty. Designed for use with Claude Code.

Installation

This entry records only its repository, not the path inside it, so there is no exact command to give. Open the source below and copy the folder into ~/.claude/skills/, or the file into ~/.claude/agents/.

README

Agentic Retrieval Experiment Framework

A framework for measuring how well AI agents retrieve information and reason over large text corpora. Generates controlled experimental corpora with hidden "needles," runs agents against them, scores their responses, and produces analysis-ready data.

Designed for use with [Claude Code](https://docs.anthropic.com/en/docs/claude-code) and the Claude Agent SDK, but the generated corpora and answer keys can be used to evaluate any agent.

Experiment Types

Type	Description
`single_needle`	Find one hidden fact in the corpus
`multi_chain`	Follow a chain of N cross-references across files to reach a final value
`multi_reasoning`	Locate N scattered clues and synthesise them to answer a question

Each type can be parameterized across:

Content profiles: e.g. Python repository, noir detective fiction
Corpus sizes: e.g. 20k, 40k, 160k, 800k tokens
Reference clarity: exact keyword, synonym, contextual paraphrase
Needle counts: e.g. 2, 8, 16 items (multi-chain and multi-reasoning)
Discriminability: easy, hard

Define your own parameter grids in `experiments/*.yaml`.

Pipeline

Generate — builds realistic background corpora from content profile templates and inserts needle payloads at controlled difficulty levels
Run — executes the agent against each corpus with a retrieval question
Judge — scores agent responses against answer keys using rubric-based LLM evaluation (correctness + completeness)
Analyse — loads verdicts into notebooks for visualisation and interpretation

Getting Started

Prerequisites

Python 3.12+
Poetry
An Anthropic API key (set ANTHROPIC_API_KEY in your environment)

Install

git clone https://github.com/mehrabmodi1/agentic-retrieval.git
cd agentic-retrieval
poetry install

Run Tests

poetry run pytest -v

Genera