Five Bugs — Testing skill for Claude Code

five-bugs

A ten-minute smoke test for AI coding agents, and the numbers it produced.

Five single-file Python bugs. One checker the agent is given. **One checker the agent never sees.** The gap between those two is the whole point.

git clone https://github.com/vyang472/five-bugs
cd five-bugs
python3 scripts/verify.py     # replays the archived patches, runs no agents, costs nothing

Everything here was run on one laptop. The tasks, the patches each agent actually wrote, the raw timings and token counts, and the scripts that produced them are all in this repo.

---

The finding I did not expect

Claude Code and Codex CLI both fixed all five bugs. Both passed the checker they were given, 5/5.

Then I wrote six new cases per bug — after the patches were in, never shown to either agent — and ran them again.

task agent    visible   hidden
t1   claude   PASS      PASS
t1   codex    PASS      PASS
t2   claude   PASS      PASS
t2   codex    PASS      PASS
t3   claude   PASS      PASS
t3   codex    PASS      PASS
t4   claude   PASS      FAIL
t4   codex    PASS      FAIL
t5   claude   PASS      PASS
t5   codex    PASS      PASS

Four of five held. On the fifth, **both agents failed in exactly the same way.** The bug was a greedy regex:

re.sub(r"<.+>", "", html)      # the seeded bug

So I ran it again on three more agents — Claude Haiku 4.5, Sonnet 5 and Opus 5. Five agent configurations, two labs, three model tiers. They wrote three *different* patterns:

agent	patch	hidden
Claude Code (default)	<[^>]+>	FAIL
Codex CLI	<[^>]*>	FAIL
Haiku 4.5	<.+?>	FAIL
Sonnet 5	<[^>]+>	FAIL
Opus 5	<[^>]*>	FAIL

Three distinct fixes. One identical wrong answer:

strip_tags("a < b and c > d")   # -> "a  d"    (all five)

None of them invented the blind spot. The visible checker contained only well-formed HTML, so that was the entire specification any of them was given