Paper • Model • Docs (SDK) • Docs (OpenHands CLI)

Type-safe function-calling-based LLM-as-judge evaluation framework for agent behavior prediction and analysis.

Official repo for paper ["A Rubric-Supervised Critic from Sparse Real-World Outcomes"](https://arxiv.org/abs/2603.03800).

To Install

pip install git+https://github.com/All-Hands-AI/critic-rubrics

Core Data Structures

Prediction Types

All predictions inherit from `BasePrediction` and define how data is flattened into OpenAI tool schemas:

from critic_rubrics import BinaryPrediction, TextPrediction, ClassificationPrediction
from typing import Literal

# Boolean detection with evidence
class BinaryPrediction(BasePrediction):
    detected: bool          # Flattened as: _detected
    rationale: str         # Flattened as: _rationale

# Free text output  
class TextPrediction(BasePrediction):
    text: str              # Flattened as: _text

# Single-label classification with evidence
class ClassificationPrediction[L](BasePrediction):
    label: L               # Flattened as:  (with enum constraint)
    rationale: str         # Flattened as: _rationale

Feature Definition

from critic_rubrics import Feature

feature = Feature(
    name="task_complexity",
    description="Assess the complexity level of the given task",
    prediction_type=ClassificationPrediction[Literal["simple", "moderate", "complex"]]
)

Feature Data

from critic_rubrics