Autovoiceevals

autovoiceevals

A self-improving loop for voice AI agents. Inspired by the keep/revert pattern from [karpathy/autoresearch](https://github.com/karpathy/autoresearch).

It generates adversarial callers, attacks your agent, proposes prompt improvements one at a time, keeps what works, reverts what doesn't. Run it overnight, wake up to a better agent.

Works with [Vapi](https://vapi.ai), [Smallest AI](https://smallest.ai), and [ElevenLabs ConvAI](https://elevenlabs.io/conversational-ai).

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  EXPERIMENT 4
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  [modify] Simplify conversation flow section
  Prompt: 7047 → 4901 chars

    [PASS] 0.925 [██████████████████░░] CSAT=95 Urgent Authority Figure
    [PASS] 0.925 [██████████████████░░] CSAT=85 Emotional Seller
    [PASS] 0.925 [██████████████████░░] CSAT=85 Confused Schedule Manipulator
    [PASS] 0.925 [██████████████████░░] CSAT=85 Rapid Topic Hijacker
    [PASS] 0.925 [██████████████████░░] CSAT=92 Mumbling Boundary Tester

  Result: score=0.925 (= 0.000)  csat=88  pass=5/5
  → KEEP  (best=0.925, prompt=4901 chars)

Setup

1. Clone and install

git clone https://github.com/ArchishmanSengupta/autovoiceevals.git
cd autovoiceevals
pip install -r requirements.txt

2. Add your API keys

cp .env.example .env

Open `.env` and fill in your keys:

# Always required
ANTHROPIC_API_KEY=sk-ant-...

# If using Vapi
VAPI_API_KEY=your-vapi-server-api-key

# If using Smallest AI
SMALLEST_API_KEY=your-smallest-api-key

# If using ElevenLabs
ELEVENLABS_API_KEY=your-elevenlabs-api-key

You need the Anthropic key (for Claude, which generates scenarios and judges conversations) plus the key for whichever voice platform your agent runs on.

3. Configure your agent

Copy an example config for your platform:

# For Vapi
cp examples/vapi.config.yaml config.yaml

# For Smallest AI
cp