adewale

Skill Eval Harness — Development skill for Claude Code

Development community

Agent Skill evaluation harness for paired variants, trace artifacts, and runner adapters.

How to install Skill Eval Harness

This entry records only its repository, not the path inside it, so there is no exact command to give. Open adewale/skill-eval-harness and copy the folder into ~/.claude/skills/, or the file into ~/.claude/agents/.

What Skill Eval Harness does

Agent Skill evaluation harness for paired variants, trace artifacts, and runner adapters.

Alternatives in Development

  • Quick Eval — Quick job evaluation 473 ★
  • Live Eval And Cost — Live Evaluation & Cost Analysis — GenAI IDP Accelerator 295 ★
  • Eval Skill — /eval-skill — Six-Dimension Evaluation 216 ★

README

Skill Eval Harness

[![CI](https://github.com/adewale/skill-eval-harness/actions/workflows/ci.yml/badge.svg)](https://github.com/adewale/skill-eval-harness/actions/workflows/ci.yml) [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)

Skill Eval Harness is a Python CLI that measures the **causal lift** of an Agent Skill: it runs the same case, model, and repetition with and without the skill, validates that exact experimental identity, then reports what changed, what passed, and whether the eval leaked its own answer. It reads `evals/shared-benchmark.json`, emits answer-key-safe task rows, grades files under `eval-runs/` locally and deterministically — no model call in the grade path — and writes benchmark reports you can diff across variants.

General eval frameworks (openai/evals, vitest-evals, viteval) score one output against a rubric. This one measures the *difference the skill makes*, and spends its surface area on keeping that difference honest: paired with/without comparison, `tune`/`holdout`/`holdback` split discipline, leakage lint, materialized ablations with provenance gates, and per-model lift. None of those frameworks have them, and they are what make a reported number trustworthy rather than merely green.

Questions this helps answer

Question Command/report to use
Does this skill improve outputs compared with no skill at all? prepare paired with_skill / without_skill rows, then benchmark paired lift and significance.
Which prompts improved, regressed, saturated, or showed no lift? benchmark case_flags, render-viewer, and error-analysis.
Could this eval have shown the lift I care about? benchmark --min-lift, then the paired lift's interval and noise_check.
Is the skill worth its extra tokens or dollars? profile-skill, token-overhead, cost-summary, and lift-per-dollar summaries.
Did my latest skill edit introduce a regression? Re-run the same manifest