Agent Belt

agent-belt

A seat belt for the agents you ship. Run reproducible multi-turn scenarios against the *binary your users actually run* - Claude Code, Cursor, Codex, Gemini, Copilot, opencode, Goose, or your own CLI. Score with rule checks, workspace diffs, and a separate LLM judge. Pin variance down with repeat trials, paraphrase families, and multi-judge consensus - so a green check actually means something on a stochastic stack.

Disclaimer

This tool is provided as-is with no warranty. Running untrusted agents or scenarios can cause real, irreversible damage to your system. Each scenario drives a real agent CLI that runs as your user - a malicious scenario can modify dotfiles, SSH config, or git hooks; run destructive or data-exfiltrating commands; or reach the network to pull instructions, upload source, or push to external repos. Only run scenarios you trust. We take no responsibility for damages.

Before You Begin

You need **Python 3.13** (3.11+ supported) and at least one CLI agent installed and authenticated (run it once interactively to sign in). One agent is enough to get started.

Install

pip install agent-belt
belt doctor   # verify agents, auth, and LLM scoring providers

For development setup, see [CONTRIBUTING.md](CONTRIBUTING.md).

Quick Start

belt quickstart              # auto-detects first available agent
belt quickstart claude-code  # or specify one

This validates the agent, runs a single scenario with rules-only scoring (no API key needed), and prints next steps.

For more control over the bundled showcase (no source clone needed):

belt eval --bundled showcase --modes rules \
  --tags real-runnable --allow-external-working-dir                # whole runnable showcase
belt eval --bundled showcase --scorer-arg model=openai/gpt-5.4-mini  # + LLM judge (cloud)
belt eval --bundled showcase --scorer-arg model=ollama/gemma4        # + LLM judge (local, no API key)
belt eval --bundled showcase --modes rule