Intent Router — AI skill for Claude Code

Intent router — a fine-tuned classifier in front of the agent

A LangGraph agent routes every incoming message before it does anything else. In my banking copilot that step was an LLM call with a structured output. This replaces it with a fine-tuned encoder and measures what changed.

Routing is closed-set classification: 77 fixed outcomes, no generation. A general model does it well, but pays a network round trip and per-token billing on the highest-QPS node in the graph — every message hits the router — and returns an answer that can differ between two identical inputs, on a step whose output is written to an audit log.

The copilot is a separate LangGraph banking agent over synthetic data and is not published; everything needed to reproduce the numbers below is in this repo.

What's here

File
train.py	Fine-tune an encoder on BANKING77 (13,083 queries, 77 intents)
bench.py	Head-to-head against the same job done by an LLM: accuracy, latency, price
calibrate.py	Pick the escalation threshold from the confidence curve, not by taste
router.py	The drop-in replacement for the copilot's route node
translate_he.py	Build a Hebrew evaluation slice — the classes are English-only otherwise
bench_llm.py	The LLM baseline on its own
negatives.py	Generate off-topic and adjacent-financial messages for a negative class
ood_eval.py, calibrate_ood.py	Score out-of-scope recall and compare models at matched false-rejection cost
hebrew_tokens.py	Token cost of Hebrew against English, per tokenizer

The design

Cascade, the same shape as the copilot's grounding check: the cheap local model settles the common case, the expensive general model runs only on what the cheap one can't account for. There the residual was arithmetic it couldn't derive; here it's confidence below threshold.

The escalation path is load-bearing, not a nicety. BANKING77 has 77 classes and every one of them is a bankin