A voice agent is not a chatbot with a phone number

Short summary

Voice agents fail when treated as chatbots with phone numbers—they require sub-500ms response budgets instead of multi-second latencies. Chat's reflex to add longer prompts backfires in voice, causing models to lose conversational context mid-call. Production voice systems use explicit state graphs with restricted tool sets per stage, not monolithic instructions, to stay focused and within latency constraints.

• •Voice systems have a 200-500ms latency budget per turn; exceeding it triggers human perceptual breakdown (silence-filling repeats, assumption of line failure)
• •Long prompts cause goal drift: models become disoriented about call stage and lose track of what they've asked, even with fast base models
• •Explicit state graphs (greeting → identity check → consent → questions → closing) with fallback handlers and restricted tool access per stage outperform single 'be helpful' prompts

Generated with AI, which can make mistakes.