The Two Bars a Voice Agent Has to Clear

The rest of this material is about how to build a voice agent that works. This one is about whether it should exist.

The argument is simple to state and uncomfortable in practice. A voice agent has to complete tasks faster and with less friction than a person, and it has to be more accurate than a person. Both, not either. Miss one and the deployment is worse than what it replaced. That, more than any technical limitation, is why adoption remains low relative to how much has been spent.

Parity is not the bar

The instinct is to aim for "as good as a human agent." That target guarantees failure, for a reason that has nothing to do with technology.

Callers extend a tolerance to people that they do not extend to machines. A human agent who mishears an account number is a moment of friction: you repeat it, they apologise, the call continues, nobody concludes anything about the company. A machine that mishears the same number is proof the system does not work. Same error, same cost in seconds, entirely different meaning to the person on the line.

That asymmetry is not irrational. People assume a machine's errors are systematic: that if it got this wrong, it will get the next one wrong too, and there is no point pressing on. They assume a human's errors are incidental. So the machine has to be enough better that the caller never has to form a theory about it.

There is also a second comparison people forget. The alternative is rarely just "a human agent." It is a human agent or the website or the app, both of which are visual interfaces with autocomplete, undo, and a list of options: the affordances a phone call does not have. A voice agent that is slower than the app and less reliable than a person is competing against both and losing to each.

Parity means the deployment was pointless. Being worse means it was harmful. The margin has to be real.

The friction bar is about completion, not speed

"Faster" is easy to misread as latency, and latency is only the visible part.

What callers experience is time to resolution: how long from dialling to the thing being done. An agent with a 400 ms response time that takes eleven turns to establish something a person would have handled in three has not made anything faster. It has made the delay smoother while making the call longer.

The bigger version of this mistake is the one that sinks the most projects. An agent that cannot complete the task is not automation. If it gathers a name, a reference number and a description of the problem, and then transfers to a human who asks for all three again, it has added a stage to the journey. The caller waited longer, repeated themselves, and arrived exactly where they would have arrived anyway. That is not a partially successful deployment; it is a net negative one, and the caller correctly perceives it as an obstacle placed between them and help.

A great deal of deployed voice AI is this. It is measured on containment (how many calls did not reach a human) while callers are experiencing deflection, which is a different thing entirely and the reason those deployments generate complaints rather than savings.

The question to ask early, and answer honestly: can this agent finish the job? Not "can it collect the information the job needs." Finish it.

The accuracy bar is really a trust bar

Accuracy on a phone call compounds in ways covered in part three: character errors multiply across a string, so per-character accuracy that sounds excellent produces field-level failure rates that are not.

But the strategic problem is not the arithmetic. It is that accuracy below the bar destroys trust non-linearly, because the caller cannot tell which interaction they are in.

An agent that is right 90% of the time is not 90% as useful as one that is right always. It is a system that must be double-checked every time, and once a caller believes that, the value is gone even on the 90% of occasions it was right. They will ask for a human, or hang up and use the app. The cost of the 10% is not 10% of the interactions; it is all of them, because uncertainty attaches to the whole channel.

This is why verifiability matters so much in choosing what to automate. Tasks where the result can be confirmed (a checksum on the reference number, a closed set of valid values, a readback the caller can affirm) allow an agent to clear the accuracy bar even when recognition is imperfect, because errors get caught inside the interaction. Tasks with no verification path require the recognition itself to be near perfect, which over a narrowband line is not a bet worth making.

The bars pull against each other

Here is what makes this genuinely hard rather than merely demanding.

Every tool for raising accuracy costs friction. Confirmations, readbacks, spelling prompts, keypad fallbacks: each one buys correctness and spends turns. Every tool for reducing friction costs accuracy. Shorter endpointing, fewer confirmations, implicit rather than explicit acknowledgement, aggressive streaming: each one buys speed and spends certainty.

A team that optimises one bar in isolation will systematically fail the other, and the failure often looks like progress: latency improved, and now the agent cuts people off; accuracy improved, and now the call takes nine turns and everyone asks for a human.

The only real escape from the trade-off is narrowness. An agent doing one well-defined thing can be accurate without much confirmation, because the space of things the caller might say is small enough to constrain, validate against, and recover from. That is why the deployments that work are almost always narrow, and why "handles anything a caller might ask" is the shape most reliably associated with failure.

Narrow and deep beats broad and shallow, every time, in this domain.

What real-time voice does to model choice

This is where cost enters, and it deserves stating plainly because the temptation runs the wrong way.

Every mitigation that makes a cheaper model workable elsewhere is unavailable on a phone call. There is no retry: the caller heard the first answer. There is no showing sources for them to check. There is no re-reading a paragraph they skimmed. There is no editing a response once audio has played, as part four covers. Batch and text applications survive weaker models because the surrounding interface absorbs the weakness. Voice has no surrounding interface.

So the quality floor for real-time voice is higher than for almost anything else you would build with the same technology, at exactly the moment your latency budget is pushing you toward smaller and faster.

That pressure produces the single most common fatal decision in this space: choosing a cheap fast model to protect the speed bar, and thereby failing the accuracy bar. The result clears one bar and misses the other, which is the same as clearing neither.

The honest framing is an economic one, and it is usually run against the wrong denominator. Inference cost should be judged against the fully loaded cost of a human handling the same contact (salary, benefits, management, floor space, training, attrition), not against API list price. Against that comparison, a materially better model at several times the price of a weaker one is still cheap, provided it clears both bars. Against list price alone, it looks extravagant, and teams talk themselves into a model that cannot do the job.

Two things follow.

If the economics only work with a model that cannot clear both bars, the project does not work yet. That is a real and acceptable conclusion. Inference costs fall steadily; the bars do not move. A capability problem today may not be one in a year.

But a task-authority problem is never solved by waiting. Which brings us to the failure that has nothing to do with models at all.

The failure nobody wants to name

The most common reason a voice agent underdelivers is not the model, the recognition, or the latency. It is that the agent was never given permission to do anything.

Read-only integrations. No write access to the system of record. Escalation required for any change, any exception, any amount above a threshold set by someone who was nervous. An agent in that position cannot clear the friction bar no matter how good its conversation is, because it structurally cannot complete tasks; it can only collect information and pass it along.

This is an organisational and integration problem wearing an AI costume, and it is expensive to discover late. It is also entirely knowable up front. Before any of the engineering in this series matters, answer: what is this agent actually allowed to do without a human? If the answer is "look things up and take messages," you are building an IVR with better speech synthesis, and it should be scoped, budgeted, and measured as one.

What working deployments have in common

The ones that succeed are not the ambitious ones. They share a shape:

• Narrow scope. One task, or a small set of closely related ones, where the space of reasonable caller inputs is small enough to constrain.
• Genuine authority. The agent can complete the task end to end, including the write, without a human in the loop.
• Verifiable results. Checksums, closed sets, confirmable readbacks: some way for errors to be caught inside the interaction rather than discovered afterwards.
• High volume. Enough repetition to justify getting narrow things very right, and enough data to know when they stop being right.
• A clean exit. When the agent cannot proceed, it hands over fast and with context, so the caller does not start again. The quality of the failure path matters nearly as much as the success path.
• Errors that are recoverable. Aggregate value high, per-error cost low. Booking a delivery slot, not authorising a payment.

Notice how little of that list is about the model.

A test before you build

Five questions, answered honestly, before any of the engineering:

1. Can it finish? End to end, without a human, for the majority of calls it will take. If not, it is a form with a voice.
2. Can the result be verified inside the call? If nothing can be checked, recognition has to be near perfect, and over a phone line it will not be.
3. Is it narrow enough to be reliable? Breadth is the enemy of both bars simultaneously.
4. Does the model that clears both bars fit the economics against fully loaded human cost? If only a weaker one fits, wait.
5. Do you know what the line is doing? The channel spends part of your accuracy budget before the model sees anything: measurably, and invisibly in your transcripts.

If the answer to any of the first four is no, the most valuable thing to do is not build it yet. That conclusion is unpopular and it is far cheaper than the alternative, which is discovering it after launch, from callers.

The technology is good enough now for a genuine set of jobs. It is not good enough for all of them, and the gap between what a demo shows and what a phone line delivers is where most of the disappointment in this category has been manufactured. Pick the jobs that fit, give the agent real power to do them, and hold both bars.

This is the strategic companion to Building Voice AI for Phone Calls, a series on the engineering problems underneath: the latency budget, why recognition fails on the alphabet, and what happens when the agent has to look something up.