lee-to

AI Tester — Testing skill for Claude Code

Testing community

End-to-end behavioral testing skills, bare system prompts, and any agent runtime — run real scenarios in an isolated git sandbox, capture the full tool-call trace, and assert it against declarative YA.

How to install AI Tester

This entry records only its repository, not the path inside it, so there is no exact command to give. Open lee-to/ai-tester and copy the folder into ~/.claude/skills/, or the file into ~/.claude/agents/.

What AI Tester does

Alternatives in Testing

Test Scenarios — Create comprehensive test scenarios from user stories 7.8k ★
Design Compare — Capture two websites with Playwright and produce a structured side-by-side design comparison — layout, color 309 ★
Baro — A CLI that turns a goal into a pull request - and a sandbox for testing concurrent AI coding agents on the Moz 120 ★

README

ai-tester

End-to-end behavioral testing for **skills**, **bare system prompts**, and agent runtimes. Run real scenarios in an isolated git sandbox, capture normalized tool-call traces, and assert behavior with declarative YAML.

[![license](https://img.shields.io/badge/license-MIT-blue.svg)](./LICENSE) [![rust](https://img.shields.io/badge/rust-1.82%2B-orange.svg)](https://www.rust-lang.org) [![CI](https://github.com/lee-to/ai-tester/actions/workflows/ci.yml/badge.svg)](https://github.com/lee-to/ai-tester/actions/workflows/ci.yml)

Why ai-tester?

LLM tests that mock the model are easy to write and weak at catching production failures. The real bugs show up in tool-use sequences, sandbox behavior, permission-mode differences, and the instructions the model actually sees.

`ai-tester` creates a throwaway sandbox per scenario, runs the selected runtime, records every normalized turn and tool call, and evaluates the run against YAML assertions.

Features

Native Rust CLI. No Node runtime or embedded SDK dependency.
Real runs, real tools. Scenarios execute inside an isolated temporary sandbox.
Multi-runtime. Built-in adapters for Claude Code and OpenAI Codex through their installed CLIs, plus generic ACP agents configured per project.
Three prompt sources. Test a packaged skill, an inline system_prompt, or an external prompt file.
Scripted user turns. Use user_prompt or user_prompts for custom session flow.
Declarative assertions. tool_called, tool_call_sequence, no_tool_called, output_contains, no_output_contains, file_read, turn_count_at_most, and no_path_escape.
Fixtures. Inline files, content_from, directory trees, staged changes, committed baselines, and setup commands.
Trace output. Every live run writes a schema 2.0.0 JSON trace under runs/.
History view. ai-tester history summarizes prior v2 traces.
Self-update. ai-tester update pulls the latest GitHub rel