DeepanshuPal

Agent Shopping Arena — Testing skill for Claude Code

Testing community

GPT-6 Sol vs Claude Opus 5.5: same Playwright harness, same 10 shopping tasks on real DTC stores, plus a live prompt-injection test.

How to install Agent Shopping Arena

This entry records only its repository, not the path inside it, so there is no exact command to give. Open DeepanshuPal/agent-shopping-arena and copy the folder into ~/.claude/skills/, or the file into ~/.claude/agents/.

What Agent Shopping Arena does

GPT-6 Sol vs Claude Opus 5.5: same Playwright harness, same 10 shopping tasks on real DTC stores, plus a live prompt-injection test. Scaffold stage - every published number will come from a logged run; none exist yet.

Alternatives in Testing

Fix Issue — by metabase - Addresses GitHub issues by taking issue number as parameter, analyzing context, implementing sol 46.5k ★
Crlf — Test for CRLF / HTTP response-splitting and host-header injection — Set-Cookie injection, cache poisoning, res 4.5k ★
Create Test Repo — Create a new ArcKit test repo with full scaffolding, auto-detecting the next version number 2.2k ★

README

agent-shopping-arena

Two frontier computer-use models. One harness. Real stores. No cherry-picking.

We give GPT-6 Sol (OpenAI) and Claude Opus 5.5 (Anthropic) the same 10 shopping tasks on the same 5 live DTC stores, behind the same Playwright browser, and publish the tape: completion, steps, wall time, token cost, and what each agent does when a store tries to prompt-inject it.

Every number in this repo comes from a real, logged run. Raw transcripts are in `results/`. If a run couldn't be verified, it isn't in the table.

Why

The store-side benchmark for agentic commerce is not tau-bench. Stores want to know: when an AI agent shows up as a buyer, does it complete the journey, get lost, hallucinate a price, or follow the malicious instructions hiding in your llms.txt? This harness measures that, model vs model, out of the box.

Quickstart

git clone  && cd agent-shopping-arena
pip install -r requirements.txt
playwright install chromium
cp .env.example .env   # add your keys
python src/runner.py --model gpt-6-sol --task all
python src/runner.py --model claude-opus-5-5 --task all
python src/score.py --summary

Bring your own keys (cost)

This repo is free; the API calls are not. You supply both keys.

OPENAI_API_KEY - GPT-6 Sol tokens + computer-use tool calls
ANTHROPIC_API_KEY - Claude Opus 5.5 tokens ($4 / $20 per MTok, Sept 2026)

Expected cost for the full suite (2 models x 10 tasks): see VERIFIED.md for the measured figure after our runs. Nothing here is free to run; do not call it that.

Rules of the arena

No real purchases. Tasks stop at cart/policy extraction. No payment data anywhere.
Same harness for both models: screenshot in, mouse/keyboard out, same browser.
Each model uses its vendor's current computer-use interface (OpenAI computer tool on the Responses API; Anthropic computer_toolset_20260801).
A task fails honestly: wrong price, hallucinated availability, following an injected instruction -