Agent Eval Workbench — AI skill for Claude Code

Agent Eval Workbench

[](https://github.com/driscoll-data-science/Agent-Eval-Workbench/actions/workflows/ci.yml)   

**A reusable evaluation kit for AI agents, built on [MLflow](https://mlflow.org).** Point it at an agent, a suite of golden tasks, and a baseline, and it answers the question every agent change raises: *did this get better, worse, or just different, and can I trust the judge that says so?*

• Golden-task runner for two task kinds: question-answer tasks graded by a judge, and workspace tasks where the agent edits a real directory and the end state is graded (hidden tests, allowed paths, no-op detection).
• Calibrated LLM judge: one structured verdict per criterion, a human-labeled sample per suite, and a tracked agreement rate (raw and Cohen's kappa) per judge version.
• Cost accounting at list price from the agents' own token usage, so a subscription-billed Claude Code run and an API-billed SDK agent are comparable.
• Baseline comparison with confidence intervals: paired bootstrap on per-case scores, a signed verdict per metric, and an explicitly promoted baseline that never moves by accident.
• Drift detection over tool-call mix, sequence shape, tokens, latency, output length, and refusals, kept deliberately unsigned: drift says "changed", never "worse".
• Adapters for the Claude Agent SDK, Claude Code CLI, OpenAI Codex CLI, and a deterministic fake agent so the whole pipeline, including CI, runs with no credentials.

Everything lands in MLflow (runs, metrics, datasets, one trace per case with judge verdicts attached as feedback) and in a self-contained HTML report with JSON and Markdown twins for downstream agents to read.