Evals Engineer — Data & AI agent for Claude Code
Use for anything under evals/ — authoring or curating golden-set tasks, implementing the outcome/trajectory/component scorers, writing and calibrating LLM-as-judge prompts against human labels, comput.
How to install Evals Engineer
Installs to ~/.claude/agents/anapsousa-agent-reliability-lab-evals-engineer.md
mkdir -p ~/.claude/agents && curl -fsSL https://raw.githubusercontent.com/anapsousa/agent-reliability-lab/HEAD/.claude/agents/evals-engineer.md -o ~/.claude/agents/anapsousa-agent-reliability-lab-evals-engineer.md Restart Claude Code, or start a new session, for it to be picked up.
What Evals Engineer does
name: evals-engineer description: Use for anything under evals/ — authoring or curating golden-set tasks, implementing the outcome/trajectory/component scorers, writing and calibrating LLM-as-judge prompts against human labels, computing pass rate / cost / p95 latency, and maintaining the CI regression gate. Trigger on "add a golden task", "write a scorer", "calibrate the judge", "why did pass rate drop", "judge/human agreement", "size the golden set", or any question about how the agent is
Alternatives in Data & AI
- GitHub Issues Specialist — GitHub Issues and Pull Requests specialist optimized for AI-agent workflows 1.5k ★
- Multi Agent Systems Architect Agent — Systems architect specializing in the design, coordination, and governance of multi-agent AI pipelines — cover 669 ★
- Peer Reviewer Computational — A simulated peer reviewer specializing in computational social science methods — NLP/text-as-data, machine lea 134 ★
Full documentation available on GitHub
View Source RepositoryRelated Agents
Judge Writer
Writes a binary LLM-as-judge prompt for exactly one failure mode, from a definition and a handful of human-lab
LLM Evaluator
LLM evaluation specialist. Use when designing eval suites for LLM features, calibrating LLM-as-judge prompts,
UI Behavior Reviewer
Compares prototype vs live UI behavior - state mismatches from the capture fingerprints, copy, labels, compone
AI Eval Engineer
Builds and maintains the eval pipeline for ai-system / agent-product archetypes. Outputs tests/eval/EVAL-.md f
Viz Frontend
Use when experiment data needs a custom dashboard, comparison surface, or trajectory viewer.
Prompt Template Engineer
Prompt template specialist for ralphctl. Use when authoring or editing any .md under src/integration/ai/prompt
Related Skills
Adversarial Testing
Use when authoring evals/predicates (Scout, TDD-Engineer), framing an audit (Auditor), or justifying which pha
Claude Evals
Production eval framework for Claude Agent SDK — implements Anthropic's published eval patterns with native SD
Sdlc Evals
/sdlc-evals — Author the Versioned Golden Set for an LLM-Powered Spec