Cc Judge

cc-judge

TypeScript-first CLI + SDK for planned Claude Code harness runs, LLM bundle judging, and `summary.md` + `results.jsonl` reports. Telemetry can fan out to Braintrust and Promptfoo through pluggable emitters.

Prerequisites

• Node.js 20.11+
• pnpm 9+
• Claude judge auth through claude auth login or ANTHROPIC_API_KEY=...
• Docker, if your harness launches Docker workloads

When `--judge-backend anthropic` is active and `ANTHROPIC_API_KEY` is not set, `cc-judge` runs `claude auth status` before `run`. Successful checks are cached for 24 hours under the user cache directory.

Install

pnpm add cc-judge

Quickstart

Create a harness-backed plan:

project: moltzap
scenarioId: EVAL-005
name: Cold outreach response quality
description: Verify the target agent responds helpfully to a first-contact DM.
requirements:
  expectedBehavior: The agent should answer coherently instead of returning an auth or runtime error.
  validationChecks:
    - Response contains non-empty text
    - Response stays on topic
harness:
  module: ../../packages/runtimes/dist/trace-capture-harness.js
  payload:
    runtime:
      kind: openclaw
    conversation:
      kind: direct
      setupMessage: Hello, can you explain how MoltZap conversations work?

Run it:

cc-judge run ./plans/**/*.yaml --results ./eval-results --log-level info

Successful and failed runs both emit:

eval-results/
  summary.md
  results.jsonl
  details/
    ..yaml

Harness Modules

The plan's `harness.module` path resolves relative to the plan file. The module must export `load(args)` as its default export unless the plan sets `harness.export`.

Minimal module shape:

import { Effect } from "effect";

export default {
  load(args) {
    return Effect.succeed({
      plan: {
        project: args.plan.project,
        scenarioId: args.plan.scenarioId,
        name: args.plan.name,
        description: args.plan.descripti