Eval Coach

Eval Coach

An Agent Skill for designing comprehensive AI evaluation strategies using Evaluation-Driven Development (EDD).

About

Created by [Bayram Annakov](https://linkedin.com/in/bayramannakov) while building [Onsa.ai](https://onsa.ai) - AI agents for B2B sales prospecting.

If you find this useful, say hi on LinkedIn!

What is Eval Coach?

Eval Coach guides you through a structured 5-step framework for evaluating LLM applications:

1. Define Success - Map business goals to measurable metrics
2. Design Dataset - Create diverse test cases (happy path, edge cases, adversarial)
3. Select Methods - Choose Automated, LLM-as-Judge, or Human evaluation
4. Plan Automation - Integrate evals into CI/CD
5. Monitor Production - Track drift and collect feedback

Installation

Claude Code / Cursor / VS Code

Copy this skill to your project's skills directory:

git clone https://github.com/BayramAnnakov/eval-coach.git ~/.claude/skills/eval-coach

Or add to your project:

mkdir -p skills
git clone https://github.com/BayramAnnakov/eval-coach.git skills/eval-coach

Usage

Invoke the skill by name or with trigger keywords:

/eval-coach

Or just mention evaluation-related topics:

• "help me create an evaluation strategy"
• "design test cases for my agent"
• "set up LLM testing"

The 50-40-10 Rule

A practical distribution for evaluation methods:

Tier	Method	Cost	Percentage
1	Automated (schema, keywords, latency)	$0.00/run	50%
2	LLM-as-Judge (quality, relevance)	$0.01-0.05/run	40%
3	Human Review	$5-50/run	10%

Templates Included

• templates/dataset.py - LangSmith dataset creation with example test cases
• templates/evaluators.py - 10 ready-to-use evaluators (automated, LLM-as-Judge, performance)
• templates/compare.py - Experiment comparison utilities

Key Insight: Silent Failures

The most dangerous failure