jfrog

Agent Belt — AI skill for Claude Code

AI community

Reproducible evaluation for AI coding agents.

How to install Agent Belt

This entry records only its repository, not the path inside it, so there is no exact command to give. Open jfrog/agent-belt and copy the folder into ~/.claude/skills/, or the file into ~/.claude/agents/.

What Agent Belt does

Reproducible evaluation for AI coding agents. Multi-turn scenarios against Claude Code, Codex, Copilot, Cursor, Gemini CLI, Goose, OpenCode, or any custom agent you plug in; verify behavior with rule checks, workspace diffs, multi-judge LLM consensus; pin reliability with pass^k variance across trials. Git worktrees, optional Docker sandbox.

Alternatives in AI

Cplt — Sandbox for AI coding agents 114 ★
Rigup.Nix — AI agents and Nix: parametrable skills/instructions and tools, packaged together in a reproducible and modular 86 ★
Creative Director Skill — AI creative director with recursive self-assessment: 20+ methodologies (SIT, TRIZ, Bisociation, SCAMPER, Synec 32 ★

README

agent-belt

A seat belt for the agents you ship. Run reproducible multi-turn scenarios against the *binary your users actually run* - Claude Code, Cursor, Codex, Gemini, Copilot, opencode, Goose, or your own CLI. Score with rule checks, workspace diffs, and a separate LLM judge. Pin variance down with repeat trials, paraphrase families, and multi-judge consensus - so a green check actually means something on a stochastic stack.

Disclaimer

This tool is provided as-is with no warranty. Running untrusted agents or scenarios can cause real, irreversible damage to your system. Each scenario drives a real agent CLI that runs as your user - a malicious scenario can modify dotfiles, SSH config, or git hooks; run destructive or data-exfiltrating commands; or reach the network to pull instructions, upload source, or push to external repos. Only run scenarios you trust. We take no responsibility for damages.

Before You Begin

You need **Python 3.13** (3.11+ supported) and at least one CLI agent installed and authenticated (run it once interactively to sign in). One agent is enough to get started.

Install

pip install agent-belt
belt doctor   # verify agents, auth, and LLM scoring providers

For development setup, see [CONTRIBUTING.md](CONTRIBUTING.md).

Quick Start

belt quickstart              # auto-detects first available agent
belt quickstart claude-code  # or specify one

This validates the agent, runs a single scenario with rules-only scoring (no API key needed), and prints next steps.

For more control over the bundled showcase (no source clone needed):

belt eval --bundled showcase --modes rules \
  --tags real-runnable --allow-external-working-dir                # whole runnable showcase
belt eval --bundled showcase --scorer-arg model=openai/gpt-5.4-mini  # + LLM judge (cloud)
belt eval --bundled showcase --scorer-arg model=ollama/gemma4        # + LLM judge (local, no API key)
belt eval --bundled showcase --modes rule