firecrawl

Benchmark Fantasy — Data skill for Claude Code

Data community

To compare how AI agents retrieve fantasy football data, we built a benchmark of 272 real search queries from agents, graded against nflverse stats and saved projections from 7 sites.

How to install Benchmark Fantasy

This entry records only its repository, not the path inside it, so there is no exact command to give. Open firecrawl/benchmark-fantasy and copy the folder into ~/.claude/skills/, or the file into ~/.claude/agents/.

What Benchmark Fantasy does

To compare how AI agents retrieve fantasy football data, we built a benchmark of 272 real search queries from agents, graded against nflverse stats and saved projections from 7 sites. It covers past-game stats, injuries and Week 6 projections, and compares Firecrawl Alexandria with web search and Flaim.

Alternatives in Data

Context Mode — Benchmark Results — Benchmarked against real outputs from popular Claude Code MCP servers, Skills, and dev tools 5.6k ★
Football World Cup — Football And World Cup Analytics Skills 280 ★
MySQL — Safe read-only SQL queries against MySQL databases 159 ★

README

FantasyBench

**Can AI agents get fantasy football data right?** 272 real questions, graded against saved sources.

The questions come from real searches made by Firecrawl users' agents. Three agent setups answer each one, and a program scores the answers against fixed answer keys. One NFL week (Week 6, 2026), one model (claude-opus-5-5), one try per question.

This repository ships a 137-question public sample (about half of every question type). The rest is held back so no setup can be tuned to it.

Questions	What is asked	Answer key
150 past games	stats, snaps, injuries and scores, Weeks 1–5	nflverse, cross-checked with Sleeper and ESPN
122 live	Week 6 projections and rankings	pages from 7 projection sites, saved before and after the answers

Results

On the full set of 272 questions:

setup	success rate	95% CI	past games	live	accuracy	median time
Alexandria (on Firecrawl)	80.5% (219/272)	75.4% – 84.8%	82.7%	77.9%	98.8%	47 s
Web search	71.7% (195/272)	66.1% – 76.7%	80.7%	60.7%	85.0%	65 s
Flaim	71.3% (194/272)	65.7% – 76.4%	79.3%	61.5%	92.6%	62 s

Success rate: share of questions answered fully right. Accuracy: share of checked values (a player's projection or rank, a table value, a past-game stat) that are correct; claims that can't be checked are left out.
Alexandria vs web search: +8.8 points (95% CI +2.9 to +14.7, p = 0.005). Vs Flaim: +9.2 points (95% CI +3.7 to +15.1, p = 0.005). Exact McNemar test, Holm-adjusted.

Full numbers: `benchmark/results/results.json`.

Install

git clone https://github.com/firecrawl/benchmark-fantasy && cd benchmark-fantasy
uv sync --extra dev        # Python 3.11+; creates .venv and installs everything, including pytest
uv run pytest -q

With pip instead: `pip install -e ".[dev]"`. Both install the `fantasybench-report` command; pip users can drop `uv run