Building Voice AI for Phone Calls
A 9-part series on the engineering problems specific to putting a conversational AI on a phone line. The latency budget and where it goes, why speech recognition fails on the alphabet, streaming a response you do not have yet, telling a real interruption from a cough, controlling hallucination with no screen to show your work on, and keeping state when a caller cuts you off mid-sentence.
Why a Phone Call Is the Hardest Place to Put an AI
Voice AI demos well on a laptop and struggles on a phone line. The constraints that change: a 3 kHz passband, a lossy real-time channel, and no screen.
The Voice AI Latency Budget: Where the Milliseconds Go
Callers notice response delay long before they notice a wrong answer. A breakdown of every stage between a caller stopping and your agent starting.
Why Speech Recognition Fails on the Alphabet
B, D, E, P, T, V and Z all rhyme, and the phone throws away what separates them. Why letters break voice agents, and how to stop depending on them.
The Retrieval Turn: When the Agent Has to Look Something Up
The one turn where streaming cannot save you. Handling dead air during a lookup, committing speech you cannot unsay, and keeping state while you wait.