Agent Config Harness

agent-config-harness

**Agent instruction files, scored and regression-tested like code.**

`CLAUDE.md`, `AGENTS.md`, `.cursorrules` and skill definitions steer how coding agents behave, and they are usually maintained the way prompts are maintained: edited by feel, never measured, quietly rotting. This repo treats them as something you can grade, track, and break a build over.

$ just eval backend-node
ℹ Scoring domain: backend-node
✔ Result written: evals/results/2026-08-29-backend-node.json
ℹ   Total: 24/25  Grade: A
ℹ   Findings (5):
    - [completeness] No build/run/verify commands. An agent cannot tell how to
      install, typecheck, lint or run the test suite before claiming a change works.
    - [clarity] Logging: `requestId` propagation is required but the mechanism is
      unspecified — AsyncLocalStorage vs an explicit context argument produces
      very different code.
    …

That is a real run against the example domain in this repo, not an illustration.

Three things make that possible, and they are the reason this exists:

A rubric with a number	Config files are scored 1–5 on clarity, conciseness, completeness, consistency and actionability. Out of 25, with a letter grade and specific findings — not a vibe.
A trend, not a snapshot	Scores are recorded per domain over time, so you can tell an improvement from a rewrite that felt productive.
Structural tests	A bats suite enforces the invariants a rubric cannot see: skill frontmatter matching its folder, required headings, unique names across a flat install namespace, no symlink silently turned back into a duplicate file.

**Requires an API key.** The eval harness shells out to the Claude CLI. Everything else — linking, health checks, the test suite — runs offline. If you only want the structure and the tests, you never need a key. And seeing the output needs nothing at all: [evals/examples/](evals/examples/) holds a real A and a real F for the s