Voice Agent Builder

voice-agent-builder

A field manual for shipping a production voice AI agent in 2026.

This repo contains two things:

1. BUILDERS_GUIDE.md — the long-form Builder's Guide. ~4,857 words. The article. Read this if you want the narrative, the architecture, the war stories, and the 90-day arc.
2. SKILL.md — the skill file. ~1,800 words. A self-contained, executable build contract that any skill-aware AI agent (Claude Code, Cursor, OpenClaw, Hermes) can load and follow. Hand this to your agent if you want the article built instead of read.

A third long-form article will be added later.

---

What this is

A voice agent is not a chatbot with a microphone bolted on. It is a real-time audio system where five components have to coordinate inside a 700ms window, or the conversation stops feeling like a conversation.

This repo is the playbook for getting that right. Not theory. Not vendor marketing. The specific decisions and tradeoffs from someone who built one, broke it three times, and rebuilt it four times in 90 days.

It covers:

• The three architectures of 2026 (chained pipeline, half-cascade, native speech-to-speech) and which to start with
• The five components every chained pipeline has (STT, LLM, RAG, TTS, function-calling) and the production failure modes for each
• The latency budget — the single non-negotiable that shapes every other decision
• The dual-agent RAG cache pattern from Salesforce AI Research's VoiceAgentRAG paper (316× retrieval speedup, 75% cache hit rate on warm turns)
• The two-checkpoint safety architecture (input guard before the LLM, output guard before TTS)
• Conversation design rules (writing for ears, not eyes)
• Evaluation — the four-layer framework, test set distribution, and the weekly review loop that compounds quality over time
• The six failure modes that will hit you in production
• The build order — twelve steps from "first call in 30 minutes" to "ship to 5% of traffic"

Who t