homefield

Benchmarks are away games. Test models on your home field.

---

A new model drops. The leaderboard says it's 6 points better. Your timeline says it's a game changer.

You switch. And on *your* codebase, with its weird build, its legacy auth module and its tests that need a running Redis, it's… fine? Worse? You honestly can't tell, because you're comparing vibes.

Public benchmarks measure public repos. Models have probably seen those repos. **None of them have seen yours.**

**homefield turns your own git history into a private benchmark.** Every past commit where someone changed code *and* added a test becomes a task: here's the commit message, make the change. The tests from that commit, hidden until the end, decide who passed.

A real run: Haiku vs Sonnet on simonw/llm

I mined [simonw/llm](https://github.com/simonw/llm)'s 2026 history: **16 real tasks**, each a commit that added a test the old code fails. 6 more candidates were dropped (flaky, or needing dependency changes). Then I ran two Claude models, one attempt per task:

$ homefield mine --since 2026-02-01 --stability-runs 2
Mined 16 task(s) into .homefield/tasks.jsonl
Dropped: reference fails 5, tests pass without the change 1

$ h