mahingbun-dev

AI Evo — Testing skill for Claude Code

Testing community

Self-hosted AI evaluation platform for LLMs & Agents — 9 eval types with AI judges, datasets, pre-eval scripts, analytics & A/B testing.

How to install AI Evo

This entry records only its repository, not the path inside it, so there is no exact command to give. Open mahingbun-dev/ai-evo and copy the folder into ~/.claude/skills/, or the file into ~/.claude/agents/.

What AI Evo does

Self-hosted AI evaluation platform for LLMs & Agents — 9 eval types with AI judges, datasets, pre-eval scripts, analytics & A/B testing. FastAPI + Vue 3.

Alternatives in Testing

Agent Skills Spec — A skill is a folder of instructions, scripts, and resources that agents can discover and load dynamically to p 1.9k ★
Inkline — by inkline - Structures development workflow using pnpm with emphasis on TypeScript and Vue 3 Composition API 1.4k ★
Adversarial Spec — A Claude Code plugin that iteratively refines product specifications by debating between multiple LLMs until a 514 ★

README

AgentEvo — Self-hosted AI Evaluation Platform

Systematically and quantitatively evaluate the response quality of LLMs and Agents — model configs, dataset management, pre-evaluation scripts, 9 evaluation types with AI judges, result analytics and A/B testing. Built with FastAPI + Vue 3 + Celery, deployable as a single Docker Compose stack.

![Home](docs/assets/00-home.png)

[![CI](https://github.com/mahingbun-dev/ai-evo/actions/workflows/ci.yml/badge.svg)](https://github.com/mahingbun-dev/ai-evo/actions/workflows/ci.yml) ![License](https://img.shields.io/badge/license-Apache--2.0-blue) ![Python](https://img.shields.io/badge/Python-3.12%2B-blue) ![FastAPI](https://img.shields.io/badge/FastAPI-0.128-teal) ![Vue 3](https://img.shields.io/badge/Vue-3-brightgreen)

[中文文档](README.zh-CN.md)

What it does

Preparation → Evaluation → Analysis, in one platform:

Module	Capabilities
Model configs	OpenAI / Anthropic compatible endpoints, custom HTTP (import from curl), runtime variable injection, streaming connectivity test
Datasets	Excel import (dynamic fields), Agent session file import, media fields, field-level validation
Script library	Pre-evaluation scripts (zip / folder packages), `model.py` contract declaration, online file-tree editing, scaffold templates
Evaluation tasks	AI judge / Agent long-session / consistency / success-rate / RAG retrieval / binary classification, batch runs, real-time progress over WebSocket
Judge engines	Claude Code judge, DeepSeek Harness judge, generic LLM judge graph; per-task evaluation dimensions
Results & analytics	Score details, run-to-run history, Excel/JSON/CSV export, trend statistics, side-by-side A/B comparison
Administration	Teams and user enable/disable (super admin), audit log, built-in user guide

First registered user automatically becomes the super admin; sessions are JWT (`Authorization: Bearer`), passwords stored as PBKDF2-HMAC-SHA256 hashes.