zhuyansen

Jev Search Rerank Eval — AI skill for Claude Code

AI community

Does a TypeSafe Jev rerank beat embedding search.

How to install Jev Search Rerank Eval

This entry records only its repository, not the path inside it, so there is no exact command to give. Open zhuyansen/jev-search-rerank-eval and copy the folder into ~/.claude/skills/, or the file into ~/.claude/agents/.

What Jev Search Rerank Eval does

Does a TypeSafe Jev rerank beat embedding search? Graded relevance eval (9,831 pairs, 164 zh/en queries) over the Agent Skills Hub catalog, with the judge-circularity bias measured.

Alternatives in AI

Firm — Convene your standing AI staff — memos on every beat, a board session without you, minutes with dissent preser 1.3k ★
Typesafe MCP — An mcp connector to evaluate anything fast and cheap 258 ★
Brick SR1 — brick is a smart AI Models router, based on complexity & capabilities extraction from the query to the models 128 ★

README

jev-search-rerank-eval

**Does a TypeSafe Jev rerank beat embedding search? Measured, with the judge-bias removed.**

A graded relevance evaluation over the [Agent Skills Hub](https://agentskillshub.top) catalog (33,047 skills, MCP servers and coding-agent tools; snapshot 2026-09-18) with 164 real Chinese / English / mixed queries, 9,831 labelled (query, skill) pairs, and a bake-off of:

ash-0.4.0 — the keyword ranker that ships in the `ash` CLI, run byte-for-byte via `ash_ranker.mjs`
bm25 — BM25 over the same metadata (latin words + CJK bigrams)
bge-m3 — dense retrieval with BAAI/bge-m3 (local) · text-embedding-3-small — dense retrieval via OpenRouter
jev-score(X@30) — a TypeSafe Jev score rerank of system X's top 30, one API call per query
rrf(…) — reciprocal-rank-fusion hybrids

Metric: **NDCG@10** on graded labels (0–3), plus MRR and P@3 (relevant = label ≥ 2), with paired bootstrap 95 % confidence intervals over queries.

TL;DR

claim	evidence
Jev as a standalone reranker does not beat a good embedding ranker.	`jev-score(bge-m3@30)` − `bge-m3` = +0.012 NDCG@10, CI [−0.013, +0.037]. Judged by an unrelated model only, it is −0.028 [−0.052, −0.004].
Jev fused with the embedding ranking is the best system, and that survives removing Jev from the judging.	`rrf(bge-m3, jev@30)` − `bge-m3` = +0.090 [+0.077, +0.104]; under LLM-only labels +0.064 [+0.052, +0.077].
The apparent Jev advantage under Jev's own labels is judge circularity, and it is measurable.	The same comparison reads +0.053 under Jev-only labels, +0.012 under merged labels, −0.028 under LLM-only labels.
The shipped keyword ranker's problem is recall, not ordering.	Reranking `ash`'s top-30 with bge-m3 makes it worse (−0.026). recall@10 of relevant items: ash 0.497, bge-m3 0.708.