The Voice AI Latency Budget: Where the Milliseconds Go

Callers forgive a lot. They will forgive an agent that misunderstands, that asks them to repeat, even that admits it cannot help. What they do not forgive is silence.

Response delay is the first thing a caller notices and the thing they judge the whole system on. It is also, unlike accuracy, immediately obvious to anyone who picks up the phone, which is why it tends to be the reason a pilot gets killed, regardless of how well the agent actually performed.

The useful way to think about this is as a budget with a hard ceiling, spent across stages you mostly control and a few you do not.

The ceiling

Human conversation has a rhythm that is startlingly consistent. The median gap between one person finishing and the next starting is around 200 milliseconds, and that holds across languages and cultures.

That number is faster than we can possibly plan a reply. Which tells you what listeners are actually doing: predicting where the sentence is going, preparing a response before it lands, and launching it on cue. Conversation is not strict turn-taking. It is two people running overlapping prediction loops.

No pipeline hits 200 ms, and that is fine; callers are more forgiving of a machine than of a person who takes too long to answer. But it sets the scale against which everything is judged:

| Delay before the agent starts | How it reads | |---|---| | Under 500 ms | Conversational. Nobody comments | | 500 – 800 ms | Noticeable but fine. Reads as considered | | 800 ms – 1.2 s | Awkward. Callers start to fill the gap | | Over 1.2 s | Broken. "Hello?" They think the call dropped |

That last row is the one that matters most, and it is specific to voice. On a screen, waiting is normal because there is a spinner telling you the system is alive. A phone call has no spinner. Silence on a phone line is indistinguishable from a dead call, and callers respond to it by talking, which arrives as an interruption, and now you have two problems.

Where the time goes

A full turn, from the caller's last syllable to your agent's first, spends time in roughly these stages. The ranges are typical rather than authoritative; measure your own.

Network transit inbound: 10 to 80 ms. Physical distance and routing. Domestic is usually tens of milliseconds; international or badly routed traffic can be far worse.

Jitter buffer: 20 to 60 ms. Deliberate delay, added to smooth out uneven packet arrival. It is a real cost and it is buying you something: without it, out-of-order packets become dropouts. On a poor connection an adaptive buffer grows, so this cost rises exactly when the rest of the call is struggling.

Endpointing: 300 to 800 ms. How long the system waits in silence before deciding the caller has finished. This is usually the single largest line in the budget, and it is discussed the least.

ASR finalisation: 50 to 200 ms. With streaming recognition much of the transcription has already happened; this is settling the last words and committing.

LLM time-to-first-token: 200 to 500 ms. More if a tool call is needed, and considerably more if that tool is a slow database or third-party API.

TTS time-to-first-audio: 100 to 300 ms. Time until the first playable chunk, not time to synthesise the whole response.

Network transit outbound plus far-end buffering: 30 to 140 ms. The same costs as inbound, charged again in the other direction.

Add the middles and you are around a second before any optimisation. The ceiling is 500 ms. That is the problem in one line.

Endpointing is where the budget actually goes

If you take one thing from this post: the silence threshold usually costs more than your model does, and it is a configuration value rather than an engineering project.

The system cannot know a caller has finished. It infers it, by waiting for silence and concluding that enough of it means the turn is over. Set that threshold at 700 ms and every single turn in every call carries 700 ms of pure additive delay, spent doing nothing, waiting to be sure.

Teams routinely spend weeks shaving 100 ms off inference while a tunable parameter contributes several times that.

The reason it is set conservatively is real, though. Shorten it and you start cutting people off mid-sentence, because natural speech is full of pauses that are not endings:

• Thinking pauses: "my account number is… uh…"
• Lists, where the gaps between items can exceed the gaps between turns
• Digit strings, which people group with pauses: "555… 0142"
• Anyone reading something off a screen or a card

Cutting a caller off is worse than being slow. They have to start again, they are now annoyed, and the interruption has consumed more time than the wait would have.

The way out is not picking a better fixed number. It is making the threshold contextual: shorter when you have asked a yes/no question and the answer is a single word, longer when you have asked for a phone number and know the caller is likely to pause between groups. If your agent knows what it just asked, it knows what shape of answer to expect, and can wait accordingly. Same pipeline, materially different feel.

Streaming changes the arithmetic

The stage list above implies the pipeline runs in sequence. It does not have to, and the difference between sequential and overlapped is the difference between a demo and a product.

Recognition streams. You do not wait for the caller to finish before transcribing. Text is arriving while they are still speaking, so by the time endpointing fires most of the work is done.

Generation streams. You do not wait for a complete response before speaking. The first clause can be going to text-to-speech while the rest is still being generated.

Synthesis streams. You do not wait for the full audio file. The first chunk plays while later ones are still being produced.

Chain those and the metric that matters stops being total processing time and becomes time to first audio. A three-second response that starts playing after 400 ms feels fast. A one-second response that starts after a second of silence feels slow. Callers experience the gap, not the total.

Two things break this, and both are worth designing around.

Tool calls stall the stream. If the agent has to look something up before it can say anything meaningful, there is nothing to stream, and the caller gets silence for exactly as long as your slowest dependency takes. This is where filler helps: a short, genuine acknowledgement ("let me pull that up") buys real time. Used honestly it works well. Used to paper over a five-second lookup it is worse than silence, because you have promised something is happening and then gone quiet anyway.

Committing early costs you flexibility. Once audio has played, it has been heard. If the model's first clause turns out to be wrong given what it generates next, you cannot take it back; you can only correct it out loud, which is expensive. Aggressive streaming trades revisability for speed.

Measure p95, not the mean

Average latency is a comfortable number that hides the problem.

Callers do not experience your mean. They experience individual turns, and one three-second stall in an otherwise brisk conversation does more damage than a consistently mediocre 900 ms, because the stall is where they say "hello?", start talking over the agent, or hang up.

So watch the tail. A pipeline averaging 600 ms with a p95 of 2.5 seconds is worse in practice than one averaging 800 ms with a p95 of 1.1 seconds, even though the first looks better on a dashboard.

The tail usually comes from a small number of places: tool calls hitting slow dependencies, cold starts on infrequently used paths, retries you are not counting, and the network having a bad minute.

The part of the budget you do not control

Everything above assumes your audio arrives promptly. Some of it will not, and that portion is not optimisable, only measurable.

Network transit, jitter and buffering are charged in both directions, and they are the parts of the pipeline you cannot engineer your way out of. Worse, they are variable in a way your own stack is not: the same path can be clean at nine in the morning and congested at two in the afternoon. An agent that feels responsive in testing and sluggish in production may have an identical pipeline and a different network.

Two specific traps.

Jitter buys latency twice. A connection with high jitter forces a larger buffer, which adds delay directly, and any packet still arriving late gets discarded, degrading the audio your recogniser is working from. You pay in both response time and accuracy, which is the subject of part one of this series.

Round-trip is not one-way. Most tools report round-trip time, and voice latency is one-way in each direction. Halve RTT for a single leg, but remember that a turn crosses the network twice, so the round-trip figure is charged in full against every response.

Before optimising a pipeline, find out what the line contributes. Our VoIP quality test reports latency, its p95, and jitter for a connection, and the results guide covers reading them. If a path is contributing 200 ms and spiking past 400, that is your budget spent before your stack has done anything, and no amount of inference tuning recovers it.

A working budget

For a target of 500 ms to first audio on a domestic call:

| Stage | Target | |---|---| | Network inbound + buffer | 60 ms | | Endpointing | 200 ms (contextual, short-answer case) | | ASR finalisation | 60 ms | | LLM to first token | 120 ms | | TTS to first audio | 100 ms | | Network outbound + buffer | 60 ms |

That is tight, and it is achievable only with streaming throughout, contextual endpointing, a fast path with no tool call, and a decent network. Any tool call blows it, which is the honest argument for acknowledging out loud before you look something up.

Most importantly: it shows where the money is. Endpointing and network together are 320 ms of a 500 ms budget (nearly two thirds) and both are usually addressed last, after the model work that costs less.

Next in this series: why speech recognition fails on the alphabet. The confusable letter sets, why they are worse on a phone line specifically, and how to design input flows that stop depending on getting them right.