sliamh11

Eval Auditor — Testing & QA agent for Claude Code

Testing & QA community

Audits an evaluation setup (benchmark, A/B test, or model comparison) for methodology errors that would invalidate results -- sampling mismatches, scope leakage, judge reliability, and baseline validi.

How to install Eval Auditor

Installs to ~/.claude/agents/sliamh11-deus-eval-auditor.md

Terminal

mkdir -p ~/.claude/agents && curl -fsSL https://raw.githubusercontent.com/sliamh11/Deus/HEAD/.claude/agents/eval-auditor.md -o ~/.claude/agents/sliamh11-deus-eval-auditor.md

Restart Claude Code, or start a new session, for it to be picked up.

What Eval Auditor does

name: eval-auditor linear_label: agent:eval-auditor description: Audits an evaluation setup (benchmark, A/B test, or model comparison) for methodology errors that would invalidate results -- sampling mismatches, scope leakage, judge reliability, and baseline validity. version: "1.0" model: sonnet

Role

Receive a description of an evaluation setup and systematically audit it for methodology errors that could produce misleading results. Prioritize errors that would cause a qualitativel

Alternatives in Testing & QA

Arbiter — You are a specialized validation agent 3.6k ★
Debugger — Debugging specialist for errors, test failures, and unexpected behavior 2.8k ★
Review Test Structure — Reviews test code for isolation, independence, readability, documentation quality, and proper setup/teardown 432 ★