Claude Marketplace Evaluator

claude-marketplace-evaluator

`cme` is a CLI for Claude Code marketplace health. It validates that skills route correctly and detects functional overlap between skills — without running expensive LLM-based eval suites.

Marketplace health is not about optimizing skill descriptions. It is about catching structural problems early: missing evals, broken routing, and overlapping skills that confuse Claude's router. `cme` runs fast, fits in CI, and fails loud.

Installation

Zero-install with `uvx` (recommended for CI):

uvx --from claude-marketplace-evaluator cme --help

Or install globally:

pip install claude-marketplace-evaluator

Commands

`cme routing`

Three-step pipeline that generates routing tests, checks coverage, and runs evals:

1. Generate — reads evals/evals.json from each skill directory, produces routing test YAML
2. Coverage check — verifies every skill has an evals file (fails if below threshold)
3. Routing eval runner — sends each prompt through the Claude Agent SDK, checks Claude routes to the expected skill

cme routing --plugins-dir plugins/

Flag	Default	Description
--plugins-dir	plugins/	Path to the plugins directory
--coverage-threshold	100	Minimum eval coverage percentage. Fails if any skill lacks evals
--threshold	95	Minimum routing pass rate percentage. Set to 0 to skip the eval runner
-j / --workers	4	Parallel workers for the eval runner
--timeout	30	Per-test timeout in seconds
--max-retries	1	Max retries on rate limit errors (exponential backoff)
--max-turns	5	Max agent turns per routing eval. Individual test cases can override this with a max_turns field in evals.json
--plugin		Glob filter on plugin name. Repeatable with OR semantics

Exit codes: `0` = all checks pass, `1` = coverage or routing threshold not met.

`cme overlap`

Detects functional overla