Fireworks Cost Benchmark — Development skill for Claude Code
Public demo benchmarks of cost per successful execution.
How to install Fireworks Cost Benchmark
This entry records only its repository, not the path inside it, so there is no
exact command to give. Open Arize-ai/fireworks-cost-benchmark and copy the folder into
~/.claude/skills/, or the file into ~/.claude/agents/.
What Fireworks Cost Benchmark does
Public demo benchmarks of cost per successful execution.
Alternatives in Development
- Evo — turns your codebase into an autoresearch loop — discovers what to measure, instruments the benchmark, then run 1.4k ★
- /remember — Save a finding or successful pattern to persistent hunt memory 1.2k ★
- Aqe Benchmark 325 ★
README
Cost per successful task
**A benchmark that measures what it actually costs to get a task *done*, not what a token costs.**
Most model comparisons quote price per million tokens. Production systems care about something different: *what did it cost to successfully complete the task?* Retries, failed tool calls, malformed outputs, and runs that grind to the token limit all cost real money that a token-price benchmark hides.
cost per successful task = total $ spent on a model across ALL attempts
---------------------------------------------
number of successful runsThis repo is a small, instrumented agent that runs real [Terminal-Bench](https://www.tbench.ai) tasks in Docker, grades each run with the task's own tests, and reports cost-per-successful-task per model. Every run is traced to [Arize AX](https://arize.com).
The finding
40 tasks × 10 models × 6 trials (2,400 runs), across four providers. Prices verified 2026-07-07, 2026-07-14, and 2026-07-16. Every one of the 400 (task, model) cells has exactly 6 trials.
| Model | Provider | Pass rate | Mean $/attempt | $/success | Retry tax |
|---|---|---|---|---|---|
| gpt-oss-120b | Fireworks (open) | 33% | $0.0178 | $0.0541 | 3.0× |
| gemini-3.1-flash-lite | 40% | $0.0255 | $0.0632 | 2.5× | |
| kimi-k2.6 | Fireworks (open) | 42% | $0.1632 | $0.3839 | 2.4× |
| glm-5.2 | Fireworks (open) | 42% | $0.2083 | $0.5000 | 2.4× |
| deepseek-v4-pro | Fireworks (open) | 39% | $0.2302 | $0.5877 | 2.6× |
| gpt-5.5 | OpenAI (frontier) | 67% | $0.4242 | $0.6363 | 1.5× |
| kimi-k3 | Fireworks (open) | 66% | $0.4410 | $0.6699 | 1.5× |
| gpt-5 | OpenAI | 41% | $0.3171 | $0.7687 | 2.4× |
| claude-sonnet-5 | Anthropic | 49% | $0.4945 | $1.0144 | 2.1× |
| gemini-3.5-flash | 23% | $0.2877 | $1.2331 | 4.3× |
Takeaways the metric makes visible:
- *The cheapest model per success has the worst pass rate.
Related Skills
Gql Performance Benchmark
Run GraphQL performance benchmarks against the Alkemio API and compare against a stored baseline
Hatch3r Benchmark
Run and analyze performance benchmarks. Compare results against baselines, identify regressions, and produce p
Benchmark
Compare skill scores against ideal benchmarks
Claude Managed Agents Sse Demo
A stdlib-only Python CLI that parses and pretty-prints Server-Sent Events from Anthropic's Claude Managed Agen
Advisor Tool Demo
Demo app for Anthropic's Advisor Tool (beta) — customer support agent comparing cost/quality tradeoffs across
MiniMax M2 Interleaved Thinking
Interactive demo showcasing MiniMax-M2's interleaved thinking for agentic workflows, demonstrating adaptive st
Related Agents
Bench Runner
Executes a11y skill benchmarks across hosted and local model families. Runs cloud/Codex/Ollama benchmark scrip
Component Benchmark Harvester
Given a component name, harvests design-system excerpts from 18 sources (design-corpora.md) and emits raw, sou
Execution Specialist
Trade execution specialist for order management, execution algorithm selection, transaction cost analysis, and
