kody-w

Token Bench — AI skill for Claude Code

AI community

Measure what Claude Code, Copilot CLI and a RAPP Brainstem spend per answer: tokens, model calls, seconds, right answers.

How to install Token Bench

This entry records only its repository, not the path inside it, so there is no exact command to give. Open kody-w/token-bench and copy the folder into ~/.claude/skills/, or the file into ~/.claude/agents/.

What Token Bench does

Measure what Claude Code, Copilot CLI and a RAPP Brainstem spend per answer: tokens, model calls, seconds, right answers.

Alternatives in AI

Jcodemunch MCP — Cut AI token costs 95%+ on code exploration 2.6k ★
Vomit — Clean up Claude 5's token vomit with a separate LLM 176 ★
Arbitrage — A Claude Code skill for token arbitrage: keep premium model tokens for judgment, dispatch all code-writing to 172 ★

README

token-bench

Measure what an AI host spends to answer the same questions: input tokens, model calls, seconds and right answers, turn by turn, for Claude Code, GitHub Copilot CLI and a [RAPP Brainstem](https://github.com/kody-w/rapp-installer).

The white paper built from these runs is **[The Context Tax](https://kody-w.github.io/token-bench/)**.

What it runs

One conversation of business questions, played through several **lanes** at once. A lane is a host plus a way of packaging the rules the host needs:

Lane	What the host gets
`cc:skill`, `cp:skill`	The rules as Agent Skills (`SKILL.md`, instructions only)
`cc:skill_script`, `cp:skill_script`	The same skills plus a bundled script that implements the rules
`cc:agentpy`, `cp:agentpy`	The same rules as RAPP `agent.py` files, served as MCP tools
`bs:agentpy`	The same `agent.py` files loaded in a Brainstem, called with `POST /chat`
`cp:brainstem_plain`	Copilot CLI with the brainstem-mcp plugin

Add `+fresh` to start a new session for every question (context cleared), and `#2`, `#3` for repeats.

The task in `tasks/long-session` has five capabilities with real rules: tiered sales commission with clawbacks, overtime pay, reorder points, late fees and double-paid invoices. All three packagings are generated from one implementation (`caps.py`), so every lane carries exactly the same logic. Answers are graded against the exact numbers, to the cent.

Run it

You need Python 3.10+, and whichever hosts you want to compare: `claude`, `copilot`, a Brainstem.

git clone https://github.com/kody-w/token-bench && cd token-bench

# 1. Each host's cost before any work: one trivial question
python3 longbench.py tasks/long-session --session baseline.json --lanes cc:skill,cp:skill

# 2. A Brainstem: start a measured copy of your installed one (your install is only read)
python3 start_brainstem.py            # leave it running; it pr