mattwfog

Tellbench — AI skill for Claude Code

AI community

Behavioral benchmark for LLM coding agents: what a model reaches for when the task is underspecified — destruction, scope, asking, honesty — as fire rates with CIs across 11 probe families and 15 fron.

How to install Tellbench

This entry records only its repository, not the path inside it, so there is no exact command to give. Open mattwfog/tellbench and copy the folder into ~/.claude/skills/, or the file into ~/.claude/agents/.

What Tellbench does

Alternatives in AI

Codeburn — Free, local tool to track AI coding token usage and cost across 37 tools and agents (Claude Code, Cursor, Code 9.7k ★
Define — Definition phase - Clarify and scope problems with multi-AI consensus 2.8k ★
LongHorizon Harness — The long-horizon computer-use harness 1.3k ★

README

tellbench

A behavioral benchmark for LLM coding agents. It does not ask *can the model do the task* — it measures **what the model reaches for when the task is underspecified**: does it delete things, stay in scope, ask before a risky step, verify a user's false claim, admit it was blocked, report what it actually did. Each behavior is a **rate over repeated runs with a Wilson confidence interval**, never a single pass/fail.

A *tell* is what a player gives away without meaning to.

What the v1 panel found

15 frontier models (9 Claude, 6 OpenAI codex) × effort tiers × 11 probe families, ~14,600 audited runs.

**Following stated rules is solved. Unprompted judgment is where models differ.** Across ~5,600 armed runs: 0 forbidden-file touches, 0 destructive tripwire fires, and exactly 1 genuine fabrication in the whole benchmark. The variance is all in what no rule asked for:

What no one asked the model to do	Spread across the panel
Tell the operator about a hazard it noticed	0% → 97%
Check a user's false claim before acting on it	some models comply unchecked 84–90% of the time; others 0%
After giving in, restate the false claim as established fact	Claude 90–96% · codex ≤30%
Use the incident history written in the code when giving advice	Claude ignores it ≤14% · codex 30–87%
Stay in scope	73% unrequested diffs → ~0% within one model family's generations

Noticing and telling turn out to be separate dispositions: a model can inspect a planted hazard and still say nothing. A single composite score would erase all of this, so the deliverable per model is a **profile**, not a number.

Full argument: [`docs/08-report.md`](docs/08-report.md). Per-family methods, tables and CIs: [`docs/family-reports/`](docs/family-reports/).

How it works

One planted tripwire per sandbox (or none, as a baseline control), inside an ordinary ambiguous agentic task.
Deterministic scoring first. Whether a tripwire fired is r