Airedale — AI skill for Claude Code

airedale

Config-driven evaluation harness for agentic LLM runs, reporting to Datadog LLM Observability Experiments.

[](./LICENSE)

`airedale` runs an evaluation matrix of **model × scenario × task** through provider-native agentic SDKs, exposes MCP servers and Agent Skills to the model, and reports every run to [Datadog LLM Observability Experiments](https://docs.datadoghq.com/llm_observability/).

---

What it does

Given two TOML files the harness:

1. Builds the matrix — every combination of model × scenario × task, optionally filtered by --model, --scenario, and --task flags.
2. Runs each cell with a provider-native agentic SDK: Anthropic models use `claude-agent-sdk` (Claude Code); OpenAI models use `openai-codex`. MCP servers and skills are passed natively — no custom tool-calling loop.
3. Scores each run with a per-criterion LLM-as-judge (RubricEvaluator). Each criterion is an independent judge call; the final score is the mean across criteria.
4. Reports to LLMObs — one async_experiment per cell, with the run's token usage, tool calls, judge scores, and gateway metadata.

Provider-native execution

Provider	Engine	LLMObs spans
anthropic	claude-agent-sdk	Native integration (ddtrace owns spans)
openai	openai-codex	Decorator spans (@agent / @llm)

When an MCP server is itself LLMObs-instrumented, its spans link back to the experiment so the tokens it consumes are rolled into the experiment's total. See [How distributed tracing works](#how-distributed-tracing-works) for details.

---

Installation

# From source (development)
uv pip in