Prompt Injection Bench

prompt-injection-bench

A hand-authored indirect-prompt-injection attack corpus (33 attacks, 10 techniques) and an honestly benchmarked comparison of four defenses on a real tool-using `claude-sonnet-5` agent — attack success rate, false-positive rate, and cost/latency, all measured, not asserted.

**[Try the interactive demo →](https://eshanya1.github.io/prompt-injection-bench/)** — click through all 33 attacks and 20 benign documents across 6 defenses, see exactly what the agent saw and how it responded. Every result replays a real recorded API run; no live calls happen on the page.

Why this exists

My [pr-review-agent](https://github.com/Eshanya1/pr-review-agent) project has a small first-pass defense against exactly this kind of attack — a flat regex tripwire in `guardrails.py`, whose own docstring is honest about its limits:

"Heuristic, not foolproof... a tripwire that raises the bar and forces escalation, not a guarantee."

That line was never actually measured. This project measures it: a real attack corpus, `naive_regex` ported near-verbatim from that same file as the baseline being one-upped, three genuinely different alternative defenses, and an eval harness that reports attack success rate *and* false-positive rate side by side — because a defense that blocks everything trivially "wins" on attack success while being useless.

Architecture

flowchart LR
  C[Attack + benign corpus\n33 malicious / 20 benign docs] --> A[Target agent\nclaude-sonnet-5, fetch_document + send_email tools]
  A --> D{Defense}
  D -->|no_defense| A
  D -->|naive_regex| A
  D -->|structural_spotlight| A
  D -->|llm_classifier\nclaude-haiku-4-5| A
  D -->|combined| A
  A --> S[Deterministic hijack scorer\ncanary text + send_email call args]
  S --> H[Eval harness\nASR / FPR / benign task success]
  H --> R[docs/index.html demo\nCOMPARISON.md]

Results

| Defense | ASR (action) | ASR (text) | FPR (adversarial) | FPR (plain) | Benign task success | |---|---