Skill Eval Harness — Development skill for Claude Code
Agent Skill evaluation harness for paired variants, trace artifacts, and runner adapters.
How to install Skill Eval Harness
This entry records only its repository, not the path inside it, so there is no
exact command to give. Open adewale/skill-eval-harness and copy the folder into
~/.claude/skills/, or the file into ~/.claude/agents/.
What Skill Eval Harness does
Agent Skill evaluation harness for paired variants, trace artifacts, and runner adapters.
Alternatives in Development
- Quick Eval — Quick job evaluation 473 ★
- Live Eval And Cost — Live Evaluation & Cost Analysis — GenAI IDP Accelerator 295 ★
- Eval Skill — /eval-skill — Six-Dimension Evaluation 216 ★
README
Skill Eval Harness
[](https://github.com/adewale/skill-eval-harness/actions/workflows/ci.yml) [](LICENSE)
Skill Eval Harness is a Python CLI that measures the **causal lift** of an Agent Skill: it runs the same case, model, and repetition with and without the skill, validates that exact experimental identity, then reports what changed, what passed, and whether the eval leaked its own answer. It reads `evals/shared-benchmark.json`, emits answer-key-safe task rows, grades files under `eval-runs/` locally and deterministically — no model call in the grade path — and writes benchmark reports you can diff across variants.
General eval frameworks (openai/evals, vitest-evals, viteval) score one output against a rubric. This one measures the *difference the skill makes*, and spends its surface area on keeping that difference honest: paired with/without comparison, `tune`/`holdout`/`holdback` split discipline, leakage lint, materialized ablations with provenance gates, and per-model lift. None of those frameworks have them, and they are what make a reported number trustworthy rather than merely green.
Questions this helps answer
| Question | Command/report to use |
|---|---|
| Does this skill improve outputs compared with no skill at all? | prepare paired with_skill / without_skill rows, then benchmark paired lift and significance. |
| Which prompts improved, regressed, saturated, or showed no lift? | benchmark case_flags, render-viewer, and error-analysis. |
| Could this eval have shown the lift I care about? | benchmark --min-lift, then the paired lift's interval and noise_check. |
| Is the skill worth its extra tokens or dollars? | profile-skill, token-overhead, cost-summary, and lift-per-dollar summaries. |
| Did my latest skill edit introduce a regression? | Re-run the same manifest |
Related Skills
Omk
Evidence-gated runner for Codex, Claude Code, OpenCode, and local coding agents. Routes tasks into scoped DAG
LiteBench
A pip-installable benchmark runner for LLMs and agents. Five minutes to your first eval.
Prompt Snapshot
Vitest-for-prompts. File-based prompt eval runner with snapshots and assertions — no SaaS, no dashboard, CI-na
Vibememo Eval
Periodic VibeMemo evaluation and capture. Runs on a loop (default 30m) to assess whether significant decisions
TS Evals
TS teaching-reflector eval-analysis mode — student evaluation analysis
Mix Eval Go
Go-based evaluation orchestrator for running Mix Eval tasks using browser automation agents
Related Agents
Skill Eval Reporter
Compares repeated paired execution results using blind A/B methodology and generates a skill effectiveness rep
Harness Prover
Running-app verifier for harness eval loops. Drives the live feature (browser, API, or CLI) and returns a bina
Oracle Evaluator
Runs the evaluation harness on candidate artifacts via pp_harness best-of-N and Borda. Invoke after a draft ha