Ai Tester

ai-tester

End-to-end behavioral testing for **skills**, **bare system prompts**, and agent runtimes. Run real scenarios in an isolated git sandbox, capture normalized tool-call traces, and assert behavior with declarative YAML.

[](./LICENSE) [](https://www.rust-lang.org) [](https://github.com/lee-to/ai-tester/actions/workflows/ci.yml)

---

Why ai-tester?

LLM tests that mock the model are easy to write and weak at catching production failures. The real bugs show up in tool-use sequences, sandbox behavior, permission-mode differences, and the instructions the model actually sees.

`ai-tester` creates a throwaway sandbox per scenario, runs the selected runtime, records every normalized turn and tool call, and evaluates the run against YAML assertions.

Features

• Native Rust CLI. No Node runtime or embedded SDK dependency.
• Real runs, real tools. Scenarios execute inside an isolated temporary sandbox.
• Multi-runtime. Built-in adapters for Claude Code and OpenAI Codex through their installed CLIs, plus generic ACP agents configured per project.
• Three prompt sources. Test a packaged skill, an inline system_prompt, or an external prompt file.
• Scripted user turns. Use user_prompt or user_prompts for custom session flow.
• Declarative assertions. tool_called, tool_call_sequence, no_tool_called, output_contains, no_output_contains, file_read, turn_count_at_most, and no_path_escape.
• Fixtures. Inline files, content_from, directory trees, staged changes, committed baselines, and setup commands.
• Trace output. Every live run writes a schema 2.0.0 JSON trace under runs/.
• History view. ai-tester history summarizes prior v2 traces.
• Self-update. ai-tester update pulls the latest GitHub rel