Superpowers Evals

Superpowers Evals

Behavioral eval lab for [superpowers](https://github.com/obra/superpowers). **Quorum** drives real coding-agent CLIs (Claude, Codex, Antigravity, Gemini, Hermes, Kimi, OpenCode, Pi, and Copilot) through a Gauntlet QA agent and grades them against scenario acceptance criteria plus deterministic post-checks.

Code, CLI, paths, and inline prose all use lowercase `quorum`; the capitalized form `Quorum` appears in headings and the actor table.

This is not a generic benchmark suite. It is an eval lab for workflow compliance: skill triggering, worktree behavior, subagent coordination, verification reflexes, review quality, and cost-shaping patterns.

Safety Model

quorum has two very different execution modes:

• Static/unit checks are safe for public CI. They run biome, tsc, and bun test. They do not call model APIs and do not launch agent CLIs.
• Live evals are trusted-maintainer operations. They launch Claude Code, Codex CLI, Antigravity CLI, Gemini CLI, Hermes CLI, Kimi Code, OpenCode CLI, Pi CLI, or Copilot CLI in permissive modes and collect raw transcripts, tool calls, filesystem state, and session logs.

Public CI must stay on the static/unit side of that line. Never add API keys, live `quorum run …` invocations, or dangerous-mode agent launches to public CI.

Live Eval Risk

Live evals run the Coding-Agent under test with broad execution power:

• Claude uses --dangerously-skip-permissions.
• Codex uses --dangerously-bypass-approvals-and-sandbox.
• Antigravity uses --dangerously-skip-permissions and authenticates with the per-run seeded copy of the operator's agy OAuth token file.
• Gemini uses --skip-trust --approval-mode=yolo; API-key auth is default, with opt-in OAuth auth for trusted local runs.
• Hermes uses --yolo and API-key auth in the run-local .env.
• Kimi uses --yolo.
• OpenCode uses --dangerously-skip-permissions.
• Pi uses explicit tool allowlists and API-key auth in a run-local con