OSS Migration Eval — Data skill for Claude Code

oss-migration-eval

**Decide — rigorously — whether to move an expensive LLM pipeline to a cheaper or open-source model.** A Claude Code skill that estimates *cost savings vs accuracy loss*, finds the most probable replacement models, and produces an honest go/no-go verdict with error bars.

The uncomfortable truth this skill is built around: **most eval "winners" are single-sample noise.** A real experiment crowned a decisive 9.00/10 winner off one generation per cell; a 5-seed rigor pass dropped it to 7.00 ± 2.0 — mid-pack in a six-way statistical tie. Within-model variance across seeds nearly equalled the entire between-model ranking. So the honest output is often *"no clear winner — here's the tradeoff, and here are the models you can safely rule out."* Saying that is a success, not a failure.

What it does

Given a recurring, expensive LLM task (lead scoring, extraction, classification, outbound copy, …), it:

1. Finds the fattest tasks worth migrating (pairs with `token-audit`).
2. Frames the accuracy metric — deterministic for structured output, a blind panel for subjective.
3. Builds a human-validated golden set on your frontier model, including hard and false-trigger trap cases.
4. Shortlists candidates live from quality (Artificial Analysis) and usage (OpenRouter) leaderboards — never a hardcoded list.
5. Sweeps each candidate provider-pinned across multiple seeds, recording measured cost + latency.
6. Grades with error bars and reports statistically tied vs clearly worse — you can only reliably name the losers, not a winner.
7. Optionally optimizes the gap with a prompt-tuning loop (train/test/val discipline; pairs with autoresearch).
8. Deploys via orchestration — the smart model orchestrates, the cheap model does the narrow task; local via Ollama; fine-tuning as a last resort.

Why it's different — the honesty guardrails

Baked in, because a naive eval