trungdlp

AI Gateway Testkit — Testing skill for Claude Code

Testing community

Open-source conformance and regression testkit for Anthropic- and OpenAI-compatible AI gateways, with stable test IDs, compatibility profiles, SDK checks, agent workflows, and shareable reports.

How to install AI Gateway Testkit

This entry records only its repository, not the path inside it, so there is no exact command to give. Open trungdlp/ai-gateway-testkit and copy the folder into ~/.claude/skills/, or the file into ~/.claude/agents/.

What AI Gateway Testkit does

Open-source conformance and regression testkit for Anthropic- and OpenAI-compatible AI gateways, with stable test IDs, compatibility profiles, SDK checks, agent workflows, and shareable reports.

Alternatives in Testing

Ext Apps — Official repo for spec & SDK of MCP Apps protocol - standard for UIs embedded AI chatbots, served by MCP serve 1.9k ★
Robotics Agent Skills — Agent skills that make AI coding assistants write production-grade robotics software 170 ★
Opencode Skill Creator — OpenCode skill for creating, testing, and optimizing other OpenCode skills 157 ★

README

AI Gateway Testkit

[![CI](https://github.com/trungdlp/ai-gateway-testkit/actions/workflows/ci.yml/badge.svg)](https://github.com/trungdlp/ai-gateway-testkit/actions/workflows/ci.yml) [![Go 1.25+](https://img.shields.io/badge/Go-1.25%2B-00ADD8?logo=go&logoColor=white)](go.mod) [![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)

**Know what your AI gateway actually supports before your users find out.**

AI Gateway Testkit (`agtk`) is an open-source, black-box conformance and regression runner for Anthropic- and OpenAI-compatible gateways. It turns compatibility claims into repeatable test profiles, stable assertion IDs, CI-friendly verdicts, and canonical reports you can compare or safely share.

![AI Gateway Testkit dashboard visualizing gateway inputs, compatibility checks, readiness charts, and verified results](docs/assets/ai-gateway-testkit-hero.webp)

gateway target + compatibility profile
                    |
                    v
           versioned test catalog
                    |
                    v
       protocol / SDK / agent checks
                    |
                    v
      PASS / FAIL / INDETERMINATE
                    |
                    v
        comparable canonical report

Why `agtk`?

Claims of OpenAI or Anthropic compatibility can mean anything from basic authentication to full tool-calling and SDK interoperability. A successful request alone does not prove that a gateway is ready for production workloads.

`agtk` gives teams evidence they can act on:

Precise failures: permanent scenario and assertion IDs such as OAI-TOOL-001/A04, instead of fragile log messages.
Explicit compatibility claims: versioned profiles for core APIs, tool calling, official Go SDKs, behavioral diagnostics, Codex, and Claude Code.
Honest verdicts: FAIL means observed incompatibility; missing or unavailable evidence becomes INDETERMINATE, never a false failure or pass.
**Regression-r