observability-oss

Eval From Trace

Name: Eval From Trace
Author: observability-oss

AI community

SOURCE https://github.com/observability-oss/progress-observability-plugin/blob/main/commands/eval-from-trace.md

Description

--- description: Build an LLM-as-a-Judge eval grounded in real traces from the Progress Observability Platform. argument-hint: [application/service name and optional symptom, e.g. "checkout-agent, wrong tool calls"] --- Use the `generate-eval` skill, Workflow A (from traces). Target: $ARGUMENTS Steps: 1. Survey the system's recent traffic with the metadata-only observability tools (`list_observations`) over the last 24–72h. 2. Choose a single failure mode and config from what the traces actua

Installation

Installs to ~/.claude/skills/observability-oss-progress-observability-plugin-eval-from-trace/SKILL.md

Terminal

mkdir -p ~/.claude/skills/observability-oss-progress-observability-plugin-eval-from-trace && curl -fsSL https://raw.githubusercontent.com/observability-oss/progress-observability-plugin/HEAD/commands/eval-from-trace.md -o ~/.claude/skills/observability-oss-progress-observability-plugin-eval-from-trace/SKILL.md

Restart Claude Code, or start a new session, for it to be picked up.