Claude Evals

claude-evals

**Native eval harness for Claude Agent SDK workflows.** Built by [Tribe AI](https://tribe.ai).

Every enterprise team building Claude-powered workflows hits the same wall: *"how do we know if a model upgrade broke something, and how do we find the cheapest model that's still good enough?"* No existing open-source tool answers this for Claude Agent SDK workflows specifically. `claude-evals` fills that gap.

What makes this different

• Native Claude Agent SDK integration — hooks into PreToolUse, PostToolUse, and SubagentStop lifecycle events, not just the final output
• Evaluates agentic behavior — tool selection, multi-step task completion, context retention, cost efficiency
• Ships with a real golden dataset — 50 calibrated test cases for contract review / document intelligence
• Implements Anthropic's published eval patterns — deterministic graders, LLM-as-judge, and human review queue from "Demystifying Evals for AI Agents"

Quick Start

pip install claude-evals
export ANTHROPIC_API_KEY=your-key

# Run the full eval suite
claude-evals run --suite datasets/contract-review --model claude-sonnet-4-6

# Run a specific category
claude-evals run --suite datasets/contract-review --category clause_extraction

# Compare two runs for regressions
claude-evals compare --baseline ./results/baseline --candidate ./results/new-run --fail-on high

From Python

import asyncio
from claude_evals.runner import EvalRunner
from claude_evals.report.terminal import print_terminal_report

async def main():
    runner = EvalRunner(
        suite_path="datasets/contract-review",
        model="claude-sonnet-4-6",
    )
    run = await runner.run()
    print_terminal_report(run)

asyncio.run(main())

The Golden Dataset

The reference dataset covers **contract review / document intelligence** — the most common enterprise AI use case. It includes 50 test ca