Fireworks Cost Benchmark — Development skill for Claude Code
Public demo benchmarks of cost per successful execution.
How to install Fireworks Cost Benchmark
This entry records only its repository, not the path inside it, so there is no
exact command to give. Open Arize-ai/fireworks-cost-benchmark and copy the folder into
~/.claude/skills/, or the file into ~/.claude/agents/.
What Fireworks Cost Benchmark does
Public demo benchmarks of cost per successful execution.
Alternatives in Development
- Evo — turns your codebase into an autoresearch loop — discovers what to measure, instruments the benchmark, then run 1.4k ★
- /remember — Save a finding or successful pattern to persistent hunt memory 1.2k ★
- Aqe Benchmark 325 ★
README
Cost per successful task
**A benchmark that measures what it actually costs to get a task *done*, not what a token costs.**
Most model comparisons quote price per million tokens. Production systems care about something different: *what did it cost to successfully complete the task?* Retries, failed tool calls, malformed outputs, and runs that grind to the token limit all cost real money that a token-price benchmark hides.
cost per successful task = total $ spent on a model across ALL attempts
---------------------------------------------
number of successful runsThis repo is a small, instrumented agent that runs real [Terminal-Bench](https://www.tbench.ai) tasks in Docker, grades each run with the task's own tests, and reports cost-per-successful-task per model. Every run is traced to [Arize AX](https://arize.com).
The finding
40 tasks × 10 models × 6 trials (2,400 runs), across four providers. Prices verified 2026-07-07, 2026-07-14, and 2026-07-16. Every one of the 400 (task, model) cells has exactly 6 trials.
| Model | Provider | Pass rate | Mean $/attempt | $/success | Retry tax |
|---|---|---|---|---|---|
| gpt-oss-120b | Fireworks (open) | 33% | $0.0178 | $0.0541 | 3.0× |
| gemini-3.1-flash-lite | 40% | $0.0255 | $0.0632 | 2.5× | |
| kimi-k2.6 | Fireworks (open) | 42% | $0.1632 | $0.3839 | 2.4× |
| glm-5.2 | Fireworks (open) | 42% | $0.2083 | $0.5000 | 2.4× |
| deepseek-v4-pro | Fireworks (open) | 39% | $0.2302 | $0.5877 | 2.6× |
| gpt-5.5 | OpenAI (frontier) | 67% | $0.4242 | $0.6363 | 1.5× |
| kimi-k3 | Fireworks (open) | 66% | $0.4410 | $0.6699 | 1.5× |
| gpt-5 | OpenAI | 41% | $0.3171 | $0.7687 | 2.4× |
| claude-sonnet-5 | Anthropic | 49% | $0.4945 | $1.0144 | 2.1× |
| gemini-3.5-flash | 23% | $0.2877 | $1.2331 | 4.3× |
Takeaways the metric makes visible:
- *The cheapest model per success has the worst pass rate.
Related Skills
Gql Performance Benchmark
Run GraphQL performance benchmarks against the Alkemio API and compare against a stored baseline
Hatch3r Benchmark
Run and analyze performance benchmarks. Compare results against baselines, identify regressions, and produce p
Benchmark Claude Pdfs
Benchmarks for Claude Code with and without the LiteParse skill, to improve skill usage and efficiency
Benchmark
Compare skill scores against ideal benchmarks
Claude Code Jev
Experimental Jev permission gate for Claude Code via OpenRouter, with reproducible latency and cost benchmarks
My Microduck Journal
A public, verifiable, continuable route to the Microduck's skills: policies trained from scratch on one consum
Related Agents
Benchmark Specialist
Use for benchmark execution, monitoring, and data collection during evaluation runs. Invoked for running tier
Eval Failure Analyzer
Analyze Logic-Lens benchmark/eval failures. Use after running content-evals, or when pointed at a skills-works
Bench Runner
Executes a11y skill benchmarks across hosted and local model families. Runs cloud/Codex/Ollama benchmark scrip
