Agent Eval Hub

AgentEvalHub

Multi-provider + cross-device reliability harness for AI agents. Runs the same agent spec against Claude / OpenAI / Gemini / Ollama and against real or mocked Android surfaces. Scores every run on task success, tool-call correctness, safety under attack, cost, latency, and regression over time.

Built as a QA portfolio project for the agentic-AI era.

---

Quick start

git clone 
cd agent-eval-hub
python -m venv .venv && source .venv/bin/activate
pip install -e .[dev]

cp .env.example .env
# fill in ANTHROPIC_API_KEY, OPENAI_API_KEY, and/or GEMINI_API_KEY
export $(cat .env | xargs)

# Run the same suite against two providers
agent-eval --suite suites/agent/tool_use.yaml --provider claude --model claude-sonnet-4-6
agent-eval --suite suites/agent/tool_use.yaml --provider openai --model gpt-4o-mini

# Compare two surfaces for answer agreement
agent-eval-cross --suite suites/agent/tool_use.yaml \
  --surface-a claude:claude-sonnet-4-6 --surface-b ollama:llama3.1

# A/B test with statistical significance
agent-eval-ab --suite suites/safety/red_team.yaml \
  --surface-a claude:claude-sonnet-4-6 --surface-b claude:claude-haiku-4-5-20251001

# Cross-surface safety parity (refusal behavior must match)
agent-eval-safety-parity --suite suites/safety/red_team.yaml \
  --surface-a claude:claude-sonnet-4-6 --surface-b device:llama3.1

# Dashboard + synthetic demo data
agent-eval-seed --db demo.duckdb
streamlit run src/agent_eval_hub/dashboard/app.py -- --db demo.duckdb

# Tests
pytest                                         # everything local (93+ tests)
pytest -m "not integration and not e2e"        # fast unit tier

---

Modules shipped

Module	What
1	Provider-agnostic agent loop (Claude / OpenAI / Gemini / Ollama)
2	RAG grounding suite + LLM-as-judge with defensive JSON parsing
3	Red-team suite (5 attack classes) + safety graders (refused, did_not_contain, did_not_call_tool)