Forensicds — Data skill for Claude Code

ForensicDS

**A benchmark for decision-grade production data science — and a measured capability gap in two frontier agents.**

Ten Harbor tasks, each a realistic piece of *inherited* data-science work. An agent receives a business memo, the messy operational warehouse behind it, the governing document that defines the decision — and an analysis someone else already did that **runs, is well documented, and is a correct computation of the wrong quantity.**

The agent has to notice, work out what is actually going on, and make a defensible business decision.

📊 **[Results and metrics →](METRICS.md)** · 📚 **[Full research dossier →](research_review/)** · 🧭 **[Start here if new →](research_review/00_READER_GUIDE.md)**

---

Headline results

	Gemini 3 Flash	Claude Opus 5.5
valid trials	31	27
passing trials	1	13
trial pass rate	3.2%	48.1%
task-level pass@3	1/10	5/9

Both arms graded by the identical frozen verifiers. 45 further attempts were infrastructure failures and are counted nowhere. [Full breakdown →](METRICS.md)

The finding is not the difficulty

On the four tasks that emit per-capability sub-scores rather than a single pass/fail bit:

capability	Gemini	Claude
reconstruct the evidence	93%	100%
frame the right question	73%	100%
validate its own work	92%	100%
produce the right number	17%	33%

Both models reconstruct the evidence and frame the right question at or near ceiling. **The decision-relevant number is the floor for both.** Claude never once mis-framed a problem and still got the number wrong two times in three. The *ordering* of difficulty is identical across models; only the severity moves.

But it is **not one failure mode.** The 24 instrumented trials fall into **11 distinct failure patterns**, including two that are mirror images — one task where the quantity is wrong and the *decisio