Mythos Bench

mythos-bench

LLM vulnerability detection benchmark — Semgrep internal, based on [Mythos Jagged Frontier](https://github.com/stanislavfort/mythos-jagged-frontier).

Two harnesses with identical output schemas for direct comparison:

Harness	Mode	Tool access
harness.py	Plain OpenRouter API calls	None — single-shot prompt
cc_harness.go	Claude Code CLI subprocess	Read / Grep / Bash

Both run the same test cases (function-level and whole-file) against the same five models and write results to the same JSONL schema.

---

Prerequisites

Both harnesses

• OpenRouter account with credits
• OpenRouter API key

`cc_harness.go` only

• Go 1.21+ (`go version`)

• Claude Code CLI installed and authenticated (`claude --version`)

  npm install -g @anthropic-ai/claude-code
  claude          # complete first-run login

• `git` (needed only for `-clone` mode)

`harness.py` only

• Python 3.14+ via [uv](https://docs.astral.sh/uv/)

  curl -LsSf https://astral.sh/uv/install.sh | sh

---

Setup

git clone https://github.com/semgrep/mythos-bench
cd mythos-bench

# Create .env with your OpenRouter key — never commit this file
echo 'OPENROUTER_API_KEY=sk-or-v1-...' > .env

For `harness.py`, sync the Python dependencies:

uv sync

For `cc_harness.go`, build the binary:

go build -o cc_harness cc_harness.go

---

Running

`cc_harness` (agentic, recommended)

# Dry run — print plan without invoking claude
./cc_harness -dry-run

# Full run, all models, all cases, 8 iterations each
./cc_harness

# Specific model and test case
./cc_harness -models anthropic/claude-opus-4-6 -cases openbsd-sack

# Whole-file mode with full repo context (slower, uses git clone)
./cc_harness -clone

# Reduce parallelism (default 10; lower if hitting rate limits)
./cc_harness -concurrency 3

# See all flags
./cc_harness -help

Key flags:

| Flag | Default | Des