Why a Phone Call Is the Hardest Place to Put an AI

Most voice AI is built and demonstrated under conditions that do not resemble a phone call at all. A good microphone, a quiet room, a wideband capture at 16 or 48 kHz, a local connection with no jitter, and a screen sitting there in case anything needs clarifying.

Then it gets connected to a phone number, and the accuracy falls off a cliff. Letters get confused. The agent talks over people. It responds to a cough. Callers repeat themselves and hang up.

The instinct at that point is to reach for the model: a better ASR, a bigger LLM, more prompt engineering. Sometimes that helps. But a significant share of what goes wrong on a phone call is not a model problem at all. It is the channel, and the channel has constraints that were fixed long before anyone thought about putting a machine on the other end.

This series is about those constraints and what to do about them. This first post is about why the phone line is genuinely a hostile environment for a listening machine.

The passband is the whole story

Start with the constraint that explains more failures than any other.

A traditional phone call carries audio from roughly 300 Hz to 3,400 Hz. G.711, still the most common codec on the network, samples at 8 kHz, which by the Nyquist limit caps what it can represent at 4 kHz. Everything above that is gone: not degraded, not attenuated, simply not transmitted.

This was not carelessness. It was an extremely good engineering decision made when bandwidth was scarce and expensive. That band carries almost all the energy of human speech and enough of the rest to keep speech intelligible. Generations of telephony were built on the finding that you can throw away most of the spectrum and humans will still understand each other.

Humans still manage because we bring enormous context to listening. We know the topic, we know the language's statistics, we know what a sentence is likely to be about, and we ask when we are unsure. We are not really transcribing; we are predicting, and using the audio to check.

The problem is which information the passband discards.

Fricatives (the s, f, th, z, v family) carry most of their distinguishing energy between 4 kHz and 8 kHz. That is above the cutoff. The acoustic detail that separates "s" from "f", or "z" from "v", is precisely the part the channel does not carry.

This is why the confusions on phone calls are so predictable. Not random errors scattered across the alphabet, but the same handful over and over: z and v, f and s, m and n, b and d and p. A model does not confuse these because it is weak. It confuses them because the evidence needed to tell them apart was removed before the audio arrived.

It is also why phonetic alphabets exist. "V for Victor" is not military theatre; it is a workaround for a limitation of voice channels that humans discovered the hard way, long before anyone had to explain it to a machine. Part 3 of this series is entirely about this problem and the ways around it.

The channel is lossy, and the repairs are fabrications

The second constraint is that a phone call is not a clean pipe even within its narrow band.

Voice travels as a stream of small packets, typically one every 20 milliseconds. Some arrive late. Some do not arrive at all. This is normal: every call has some of it, and the whole apparatus of VoIP quality measurement exists because of it.

What matters for a listening machine is what happens next. When a packet goes missing, the decoder does not play silence. Packet loss concealment generates plausible audio to bridge the gap: extending the previous waveform, interpolating, synthesising something that sounds like what probably should have been there.

For a human listener this is a gift, and it works remarkably well. For a model it is something else: the audio your recognition system receives contains segments that were invented by a codec. They are not a recording of what the speaker said. They are a codec's best guess, optimised to be inaudible to a person rather than to be accurate.

Jitter compounds this. Packets arriving out of schedule get buffered and reordered, and any that arrive past their playout deadline are discarded and concealed as though they had been lost. A connection with no measured packet loss at all can still be feeding your model concealed audio, because late is the same as lost once the buffer has moved on.

The practical consequence: call quality is upstream of model accuracy, and it is invisible in your logs. A transcript full of errors looks identical whether the model misheard clean audio or heard exactly what it was given and the audio was corrupt. If your agent's accuracy varies by time of day, by caller, or by carrier, that is not a model that has become unreliable; that is a channel that varies, which is exactly what channels do.

There is no screen

The third constraint is the one that changes product design rather than engineering.

Every interface convention that makes machine mistakes survivable is missing on a phone call. No autocomplete offering the right spelling. No list of options to pick from. No display of what was understood before acting on it. No undo. No source link to check a claim against. No way to show your working.

Text interfaces are forgiving because they are cheap to correct. A wrong autocomplete is a glance and a keystroke. On a phone call, every correction costs a full conversational turn, and turns are expensive: the agent has to finish speaking, the caller has to interrupt or wait, the correction has to be recognised, and the agent has to recover. What is a half-second annoyance on screen is five to ten seconds of awkwardness on a call.

This changes what "good" means. On screen, a system that is right 95% of the time and easy to correct is pleasant. On a phone call, the same accuracy with expensive corrections is exhausting, because the 5% dominates the experience. Designing for voice means designing for the correction path first, which is the opposite of how most interfaces get built.

It also means hallucination is more dangerous here, not less. On screen a confident wrong answer sits next to a link the user might follow. On a call it is a voice stating a fact, with nothing to check it against and no record afterwards. Part 6 covers what to do about that.

Conversation is a protocol nobody wrote down

The last constraint is social, and it is the one that makes agents feel wrong even when every component is working.

Human conversation runs on timing rules that are remarkably tight and almost entirely unconscious. Typical gaps between turns are around 200 milliseconds. That is faster than we can plan a response, which means listeners are predicting the end of your sentence and preparing to speak before you have finished. A silence of a full second reads as hesitation, confusion, or disagreement: it is interpreted, not merely noticed. This timing threshold is one of the two bars a voice agent has to clear; accuracy is the other.

Layered on top of that: people interrupt, and the interruptions are not all the same thing. Some are genuine attempts to take the floor. Many are backchannel ("mm-hmm", "right", "okay"), which mean keep going, the opposite of an interruption. A system that treats every incoming sound as a turn attempt will stop talking constantly. A system that ignores them all will talk over people who are genuinely trying to correct it. Telling these apart on a noisy line, with the agent's own voice echoing back, is Part 4.

And when an interruption does land legitimately, the agent has a state problem most implementations get wrong. Its text-to-speech was cut off mid-sentence, so what it generated and what the caller actually heard are now different things. If its context records the full sentence it intended to say, every subsequent turn is built on a false premise about what the other party knows. That mismatch is Part 5, and it is the most under-documented problem on this list.

What this series covers

Nine parts, working from the physical channel upward:

1. Why a phone call is the hardest place to put an AI: this post
2. The latency budget: where the milliseconds actually go, and what callers notice
3. Why speech recognition fails on the alphabet: the confusable sets, and how to design around them
4. The retrieval turn: streaming a response you do not have yet, and what happens to state while you wait
5. Barge-in: separating a real interruption from a cough, a backchannel, or your own echo
6. What the caller actually heard: keeping state when your own speech gets cut off
7. Hallucination control without a screen: grounding and confirmation when there is nothing to show
8. Changing instructions mid-call: handling "actually, make that Tuesday" without losing the thread
9. Testing a voice agent: measuring something non-deterministic over a channel that varies

A separate strategic companion follows the series, on whether a given voice agent project should exist at all: an agent has to be both faster and more accurate than a person, parity is a failing grade, and the deployments that work share a recognisable shape.

Before you tune anything, know what the line is doing

One practical note to close on, because it saves a category of wasted effort.

If a voice agent is performing worse in production than in testing, establish what the audio path is doing before changing anything about the model. Packet loss, jitter and concealment produce accuracy problems that look exactly like model problems in the transcript, and they are not fixable at the model layer: the information is already gone.

That is measurable. Our VoIP quality test reports loss, jitter, latency and a standards-based MOS score for a connection, and the results guide explains what the numbers mean. If a call path is scoring poorly, your model is working from damaged input, and that is where to start.

Next in this series: the latency budget, where the milliseconds go between a caller finishing a sentence and your agent starting one, and how much of it callers actually notice.