Eval Task Auditor — Project Management agent for Claude Code
Audits eval task files for the problems that most often break real benchmarks: ambiguous success criteria, wrong reference solutions, graders that are too strict or too loose, and shortcuts that pass.
How to install Eval Task Auditor
Installs to ~/.claude/agents/jddavenportopen-agent-evals-starter-eval-task-auditor.md
mkdir -p ~/.claude/agents && curl -fsSL https://raw.githubusercontent.com/jddavenportOpen/agent-evals-starter/HEAD/.claude/agents/eval-task-auditor.md -o ~/.claude/agents/jddavenportopen-agent-evals-starter-eval-task-auditor.md Restart Claude Code, or start a new session, for it to be picked up.
What Eval Task Auditor does
name: eval-task-auditor description: Audits eval task files for the problems that most often break real benchmarks: ambiguous success criteria, wrong reference solutions, graders that are too strict or too loose, and shortcuts that pass without doing the work. Use after drafting or changing tasks, and before trusting a new suite. tools: Read, Grep, Glob, Bash
You audit eval tasks. You do not fix them; you report, with evidence, so a human decides.
For each task file you are given (or e
Alternatives in Project Management
- Qualityqueen — Quality Assurance & Code Standards Specialist who ensures code meets project standards across any technology s 38.1k ★
- Requirements Analyst — Transform ambiguous project ideas into concrete specifications through systematic requirements discovery and s 1.5k ★
- Orchestrator Agent — You are a senior software architect and project coordinator 1.3k ★
Full documentation available on GitHub
View Source RepositoryRelated Agents
Fairness Verifier
Reviews and executes a software-engineering task package's verifier — runs the oracle solution, a no-op, and p
Delegate Advisor
Top-tier escalation for the /delegate protocol. Read-only. Called at most once per run, at the single moment t
Documentation Auditor
An expert documentation specialist that comprehensively audits and updates project documentation for accuracy,
Ste Glossary Auditor
Audits a documentation set and the code for terminology drift, then proposes a project glossary and the matchi
Reality Checker Tram
Tier-2 Harness Auditor for Team RAM. Cross-session, cross-project health audits. Verify invariants are enforce
Wave Auditor
Independent auditor for one feature-wave stage. Did not write the code. Checks the stage's diff against its ac
Related Skills
Cheat Index
The Agent Cheat Index: how often coding agents edit tests, special-case inputs or claim success they did not e
AI To Agent Custom Skill
AI to Agent 上位決策治理 Skill — Context over Dogma, Problems over Benchmarks, People over Models, Outcome over Comp
Evomap
Connect to the EvoMap collaborative evolution marketplace. Publish desensitized error fixes from the local led