Harness Bench — Development skill for Claude Code

HarnessBench

HarnessBench compares coding-agent harnesses on real-repository debugging tasks.

The benchmark asks each harness to fix the same bug from the same sanitized repository checkout, then scores the patch with hidden behavioral tests. The current scope is Codex, Claude Code, Cursor Agent, and Antigravity CLI, but the condition schema is harness-neutral.

Project name:

HarnessBench: Comparing Coding Agent Harnesses on Real-Repository Debugging Tasks
Repository: nyosegawa/harness-bench

Design

HarnessBench uses a two-layer scoring model:

• core_tests: the observable bug-fix contract. Every core test must pass.
• regression_tests: targeted surrounding behavior that must not break. Every regression test must pass.

Official cases use:

test_strategy:
  core_tests:
    - benchmark/cases//hidden-tests//core.sh
  regression_tests:
    - benchmark/cases//hidden-tests//regression.sh
  success_rule: core_and_regression

Core tests define the required user-visible behavior as a behavioral contract. Regression tests protect nearby behavior that should remain unchanged.

Artifact Policy

`benchmark/runs/`, `benchmark/workspaces/`, and `benchmark/archive/` are local data and are ignored by git. Raw logs are preserved under `benchmark/archive/` when experiments are reset.

Official, publishable experiment artifacts live under:

benchmark/experiments//
  manifest.json
  summary.json
  failure-reviews.json
  results.html

The public report index is:

benchmark/reports/index.html

After a reset, old experiments are archived locally and are not part of the public benchmark record.

Citation

Use the repository citation metadata in `CITATION.cff`. A paper citation can be added there after the manuscript is public.

Quick Checks

node --check scripts/run-case.mjs
node --check scripts/run-matrix.mjs
node --check scripts/render-results.mjs
node --chec