Octobench

octobench

[](https://github.com/Muvon/octobench/actions/workflows/ci.yml) [](LICENSE)

**WIP Notice:** This project is actively under development and APIs/behavior may change without notice. Use at your own risk while we stabilize it.

Benchmark framework to compare **LLM tool + config + prompt** setups across a shared set of cases.

**Latest results:** see [BENCHMARK.md](BENCHMARK.md) — the real-commit benchmark (80 fail-to-pass-proven one-shot tasks and 25 long-run sequences harvested from merged pull requests in trusted OSS projects across 5 languages) comparing claude, codex, octomind, and opencode, with full reproduction instructions.

Contribution guide: see [CONTRIBUTING.md](CONTRIBUTING.md) (focused on adding new cases).

Note: `octomind` is Muvon's own coding agent — we benchmark it under the same fairness rules as every other client.

Documentation

• docs/USAGE.md — CLI reference, executors, env vars
• docs/ARCHITECTURE.md — concepts and case lifecycle
• docs/EXTENDING.md — adding cases and benchmarks
• docs/PROVIDER_INTERFACE.md — provider contract
• docs/HARNESS.md — real-commit case construction rules
• docs/ONESHOT.md / docs/LONGRUN.md — case formats
• AGENTS.md — onboarding entrypoint for contributors and agents

Key ideas

• Cases define prompts and scripts.
• Providers are Python implementations that run tools and return normalized telemetry.
• Judge is an LLM prompt with strict JSON output.
• setup.sh / quality.sh / validate.sh are bash scripts whose logs are fed to the judge.

Quick start

1. Create a venv and install deps (Python 3.11+):

python3 -m venv .venv
source .venv/bin/activate
pip in