yangstar89

Ground Truth Evals — AI skill for Claude Code

AI community

An LLM eval harness graded by computed ground truth, not an LLM judge.

How to install Ground Truth Evals

This entry records only its repository, not the path inside it, so there is no exact command to give. Open yangstar89/poker-agent-evals and copy the folder into ~/.claude/skills/, or the file into ~/.claude/agents/.

What Ground Truth Evals does

An LLM eval harness graded by computed ground truth, not an LLM judge. Worked example: poker, served to models as MCP tools — gpt-4o-mini goes from 26/75 to 75/75 with them.

Alternatives in AI

Cc Mini — Ultra-light Harness scaffolding for AI agents, a mini version of claude code 825 ★
Base — AI builder operating system 172 ★
Rails AI Context — 45 MCP tools that give AI coding agents ground truth about your Rails app: schema, models, routes, controllers 155 ★

README

ground-truth-evals

[![tests](https://github.com/yangstar89/ground-truth-evals/actions/workflows/test.yml/badge.svg)](https://github.com/yangstar89/ground-truth-evals/actions/workflows/test.yml) [![license: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)

An evaluation harness that scores an LLM's poker decisions against **computed ground truth** rather than a rubric or another model's opinion.

Most LLM evals grade fuzzy work — is this summary good, is this answer helpful — and end up leaning on a human rubric or an LLM judge, both of which are noisy and neither of which is cheap to re-run. Poker is arithmetic. `AcAd` against `KcKd` on a `2c 7d 9h` flop has an exact equity, reachable by enumerating every remaining board. That makes every grader here deterministic, every run reproducible, and re-grading free.

The harness itself knows nothing about poker: `src/` runs models, caches replies, grades, diffs and reports, and a *suite* supplies the domain. Poker is the worked example; `examples/dates/` is the same harness over date arithmetic in eighty lines. To point it at your own domain, read **[docs/harness.md](docs/harness.md)**.

Results

The same 75 cases, the same prompt, each model run unaided and then with the oracle available as MCP tools. Cases passed:

	equity	icm	range	all
gpt-4o-mini, unaided	3/31	0/16	23/28	26/75
gpt-4o-mini, with tools	31/31	16/16	28/28	75/75
claude-sonnet-5, unaided	19/31	14/16	25/28	58/75
claude-sonnet-5, with tools	31/31	15/16	28/28	74/75

What the runs show:

Unaided, the errors are large, not marginal. gpt-4o-mini's equity answers were 17.5 percentage points out on average. Re-scored at double the tolerance (5pp), it passes 8 of 31 instead of 3; Sonnet 5 passes 20 instead of 19. The thresholds are not what decides these numbers.
The two models fail differently. gpt-4o-mini answers at once and is w