djmoore-projects

Agent Reliability Harness — Git skill for Claude Code

Git community

Evals, tracing and CI regression gates for tool-using agents: tool-selection, final-state, cost, latency and step metrics, pass^k reliability, fault injection, and a PR gate that fails when agent qual.

How to install Agent Reliability Harness

This entry records only its repository, not the path inside it, so there is no exact command to give. Open djmoore-projects/agent-reliability-harness and copy the folder into ~/.claude/skills/, or the file into ~/.claude/agents/.

What Agent Reliability Harness does

Evals, tracing and CI regression gates for tool-using agents: tool-selection, final-state, cost, latency and step metrics, pass^k reliability, fault injection, and a PR gate that fails when agent quality drops.

Alternatives in Git

  • Cc Usage — by ryoppippi - Handy CLI tool for managing and analyzing Claude Code usage, based on analyzing local Claude Co 11.7k ★
  • Skill Copilot Provider — GitHub Copilot CLI as optional zero-cost provider via copilot -p programmatic mode 2.8k ★
  • Droidconkotlin — by touchlab - Delivers comprehensive Gradle commands for cross-platform Kotlin Multiplatform development with 1.1k ★

README

agent-reliability-harness

[![CI](https://github.com/djmoore-projects/agent-reliability-harness/actions/workflows/ci.yml/badge.svg)](https://github.com/djmoore-projects/agent-reliability-harness/actions/workflows/ci.yml) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE) [![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/downloads/)

**Evals, tracing and CI regression gates for tool-using agents.** Point it at a task suite and it answers the question every agent project eventually gets asked — *"how do we know it works?"* — with numbers: did the agent pick the right tools, is the final state (and the real-world side effect) correct, what did each task cost, how long did it take, how many steps did it need, and does it still pass when an API falls over halfway through. Then it runs as a CI check that fails the PR when any of that gets worse.

It's the agent-shaped sequel to [fintech-rag-eval](https://github.com/djmoore-projects/fintech-rag-eval): that project grades RAG *answers*; this one grades agent *runs*. Same philosophy — typed contracts, structured judge outputs, scores computed in Python rather than reported by a model.

First real target: b2b-account-pipeline

The harness was built against [b2b-account-pipeline](https://github.com/djmoore-projects/b2b-account-pipeline), a LangGraph agent that researches a company with Claude + Tavily, writes a 3-step outreach sequence and syncs a CRM record. Its README says it "works" and shows a clean run. The 11-task suite ([`evals/suite.yaml`](https://github.com/djmoore-projects/b2b-account-pipeline/blob/agent-reliability-evals/evals/suite.yaml)) checks only the contract the pipeline states for itself, and the checks were frozen before the baseline ran.

**Baseline: 17% task success** (3/18 clean trials, 95% CI 6–39%). **All 13 clean happy-path trials failed:**

  • [Name] / [Your Name] placeholders in outreach that is meant to be sent as-is: