Eval Bench — AI skill for Claude Code

eval-bench

Benchmark Claude Code plugins by A/B comparing plugin versions with LLM-judged evaluation prompts.

Runs a fixed set of prompts against two versions of your plugin (baseline vs current), invokes the real `claude` CLI so skills, MCP servers, subagents, slash commands, and hooks actually load, grades each output with a configurable judge (local Ollama, the `claude` CLI itself, Anthropic, OpenAI, OpenRouter, GitHub Models, or any OpenAI-compatible endpoint), and produces a side-by-side comparison.

How it works

sequenceDiagram
    actor User
    participant eb as eb CLI
    participant git
    participant claude as claude CLI
    participant judge as Judge
    participant snap as snapshot.json

    User->>eb: eb run --baseline 
    eb->>git: worktree add baseline
    git-->>eb: baseline dir (current side uses the working tree)

    Note over eb,judge: Per row: claude then judge, interleaved (runs.parallel controls concurrency)
    loop prompt × {baseline, current} × samples
        eb->>claude: spawn with plugin dir + prompt
        claude-->>eb: stdout
        eb->>judge: invoke {prompt, output, rubric}
        judge-->>eb: score 0–5 + rationale
    end
    Note right of eb: with --baseline-from, baseline runs are reused from a saved snapshot. --current-from does the same for the current side.

    eb->>snap: write runs + judgments + stats
    eb-->>User: eb compare → markdown / json · eb view → HTML

The provider (`claude` CLI) and judge are independent — the judge never sees `claude`, only the captured output and your rubric. Judges are pluggable (Ollama, Anthropic, OpenAI, OpenAI-compatible, OpenRouter, GitHub Models, or the local `claude` CLI as judge); most are HTTP-backed, `claude-cli` shells out.

Install

npm i -g eval-bench
# or
npx eval-bench --help

Requires:

• Node 20+
• claude CLI on PATH (install instructions)
• Your plugin in a git repo (required for baselin