Open Agent Octagon

OpenAgentOctagon

**English** | [中文](#openagentoctagon-中文)

**A same-task, multi-agent comparison harness.** Give the same batch of tasks to several coding / model agents, and compare their execution, reasoning, and final artifacts side by side in one UI.

Octagon (the UFC cage) — agents compete on the same stage, differences laid bare.

One run, seven agents on the same SWE-bench task — scores, cost, and behavior compared column by column.

OpenAgentOctagon is a **comparison tool**, not a leaderboard. Its core output is "the visualized difference between several agents doing the same thing"; scoring is only a quantitative anchor for locating those differences. Each comparison captures three things:

1. Execution — tool calls, errors, retries, timing
2. Reasoning — task understanding, planning, decision forks, self-correction
3. Final artifacts — business state, code files, test results, scores

Stack: Python 3.12+ / FastAPI / SQLite / uv (backend) + React + Vite + TypeScript (frontend).

---

Framework vs. scenarios

This repo is the **framework**: dispatch, execution, observation, scoring, frontend, and a stable scenario interface (`octagon.env_api`). It ships **no real evaluation scenarios** — only two minimal examples (`envs/example-tool-use`, `envs/example-coding`) for quickstart and contract reference.

Real evaluation **scenarios** (envs) are separate assets, kept in their own repo and mounted via the `envs_path` config. The framework has zero hard dependency on any scenario: at runtime it scans the `envs_path` directory and dynamically loads each scenario's `meta.yaml` / `core.py` / `scorer.py` / `tasks/`.

┌──────────────┐   envs_path points to   ┌────────────────────┐
│ OpenAgent    │ ──────────────────────► │ your scenario repo  │
│ Octagon      │