Judge Writer — Data & AI agent for Claude Code
Writes a binary LLM-as-judge prompt for exactly one failure mode, from a definition and a handful of human-labeled examples, and sets up its validation against held-out human labels.
How to install Judge Writer
Installs to ~/.claude/agents/jddavenportopen-agent-evals-starter-judge-writer.md
mkdir -p ~/.claude/agents && curl -fsSL https://raw.githubusercontent.com/jddavenportOpen/agent-evals-starter/HEAD/.claude/agents/judge-writer.md -o ~/.claude/agents/jddavenportopen-agent-evals-starter-judge-writer.md Restart Claude Code, or start a new session, for it to be picked up.
What Judge Writer does
name: judge-writer description: Writes a binary LLM-as-judge prompt for exactly one failure mode, from a definition and a handful of human-labeled examples, and sets up its validation against held-out human labels. Use when a failure mode can't be checked with code. Refuses to label the validation data itself. tools: Read, Grep, Glob, Write, Bash
You write LLM judges that a team can trust because they were checked against people.
Inputs you need before writing anything: the failure mod
Alternatives in Data & AI
- Gn Voice Judge — gn-voice 윤문 결과물의 문체 품질을 판정하는 심사 에이전트 93 ★
- Leptos Designer — Use this agent when designing or implementing UI components for ccboard's web frontend (Leptos WASM) 92 ★
- Analytics Analyst — Invoke when the user needs help with marketing measurement, KPI definition, dashboard design, attribution mode 64 ★
Full documentation available on GitHub
View Source RepositoryRelated Agents
Budgeter
Resource gatekeeper. Other agents must call the budgeter before launching long-running training, expensive LLM
Data Builder
Builds and audits datasets for tiny models — synthetic generation, teacher labelling, held-out splits, and lab
Sf Metadata Engineer
Implements declarative Salesforce metadata - custom objects and fields, permission sets, Flows, layouts, Flexi
Evals Engineer
Use for anything under evals/ — authoring or curating golden-set tasks, implementing the outcome/trajectory/co
Autoresearch LLM Evaluator
Semantically judge AutoResearch task trajectories and produce structured failure-mode analysis.
Review Arbiter
Authoritative LLM-as-a-Judge meta-agent that adjudicates multi-agent plan and code reviews whenever two or mor
Related Skills
Openbles Firewall
OpenBles Firewall 1: an open 31.8 MB decision model that catches prompt injection in what an AI agent reads (w
Code Writer
You generate exactly one new file from the instructions and the attached
Judge Initial
$magnum-review Review {{.URL}} and post exactly one GitHub review as {{.ReviewerLogin}}.