The Retrieval Turn: When the Agent Has to Look Something Up

Every other turn in a conversation can start fast. The model has context, it can begin generating immediately, and streaming hides most of the work behind the first few words.

Then the caller asks something the agent does not know. What is my balance? Has it shipped? Do you have that in stock on Thursday?

Now the pipeline has to stop. There is no response to generate, because the facts the response depends on are somewhere else: a database, an internal API, a third party with an SLA that was never written with phone calls in mind. Every technique that made the other turns fast relies on having something to say, and here you have nothing.

This is the hardest turn in a voice agent, and it is the one most likely to be designed last.

You cannot stream what you do not have

Worth being precise about why streaming stops helping, because the instinct is to reach for it harder.

Streaming works by overlapping stages that are all running: transcription happens while the caller speaks, generation begins before transcription finalises, synthesis begins before generation completes. Every stage is fed by a stage already producing output.

A retrieval turn breaks that chain at the source. The generator has no input, because the retrieval has not returned. You cannot overlap a stage with one that has not started, and no amount of streaming infrastructure changes that. Until the data arrives, the honest state of the system is: nothing to say yet.

Meanwhile the caller is sitting in silence, and from part two we know how that reads. There is no spinner on a phone call. Past about a second and a half of nothing, callers assume the line has dropped and start talking, which lands as an interruption, on top of a turn that was already struggling.

The gap also has a length you do not control. Your own inference times are yours to optimise. A third-party availability check is not.

Say something, and be careful what

The fix is to fill the silence with something honest. That is straightforward in principle and has three traps in practice.

The acknowledgement is committed before the answer is known. You decide to say "let me check that for you" while the lookup is still in flight. Whatever comes back (success, empty result, timeout, error), that sentence has already been heard. So it has to be true in every branch.

This makes one specific mistake worth naming: never presuppose the outcome. "Let me pull up your order" is safe. "I've found your order, one moment" is not, because you have not, and if the lookup returns nothing you now have to contradict yourself. Callers notice that, and it costs more trust than the original delay would have.

Say that you are looking. Never say that you have found.

Filler has to be latency-aware. If the lookup returns in 200 ms, an acknowledgement makes the turn slower and the agent chattier than it needed to be. If it takes four seconds, silence would have been fatal. So the decision to speak should be conditional: emit the acknowledgement only when the call has not returned within some threshold, typically a few hundred milliseconds.

Mind the double gap. This one is subtle and common. The agent says "let me check that" in 800 ms, the lookup takes three seconds, and the caller now experiences silence, a brief interjection, and more silence. Two gaps with a fragment between them is worse than one continuous gap, because the first one raised an expectation the second one broke.

If a dependency is reliably slow, the acknowledgement needs to be long enough to cover a meaningful part of the wait, or the agent needs something else to do: asking the next question it will need anyway is far better than narrating its own progress twice.

Vary the wording. A caller who hears the identical phrase three times in one call has learned it is a recording, and the illusion that the agent is paying attention does not survive it.

Start earlier than the question ends

The best fix for a slow lookup is starting it sooner.

Intent is often clear well before a sentence finishes. "Can you tell me what my balance…" is unambiguous by the third word. If your pipeline is running streaming recognition, you have partial transcripts arriving continuously, and there is nothing stopping you dispatching the account lookup while the caller is still talking. By the time endpointing fires, the data may already be sitting there, and a turn that would have needed filler becomes indistinguishable from a normal one.

Two conditions on this, and they are not optional.

Only for reads, and only for idempotent ones. Speculative execution of anything that changes state is a bug generator. Fetching a balance early is free. Cancelling an order early is not.

Budget for waste. You will fire lookups for turns that end up going elsewhere. If the dependency is rate-limited, metered, or expensive, that waste is real and needs accounting for. Cheap internal reads are the obvious candidates; per-query billed third parties usually are not.

Committing speech you cannot unsay

Here is the part that makes streaming into a voice generator genuinely different from streaming onto a screen.

Text on a screen can be revised. Speech cannot. A streaming chat response that corrects itself mid-paragraph is barely noticed: the earlier tokens are still on screen, the reader integrates the correction, no harm done. Audio that has played has been heard. There is no editing it, only contradicting it out loud, which costs a turn and sounds like the agent is confused.

This matters most on exactly the turn we are discussing, because a retrieval turn is where the model is reasoning over freshly-arrived data and its early tokens are least reliable. The model starts generating from a partial view, and later reasoning can reverse the direction of the sentence it already began.

The failure looks like this: the agent starts saying "Your order shipped on…" while the reasoning is still resolving which of three orders the caller means. Now either it stops mid-sentence, or it finishes a sentence it no longer believes.

The workable rule is to treat different kinds of text differently:

Stream framing freely. "Right, I've got your account here." "Okay, so…" Connective and social language carries no factual commitment and can go to synthesis the moment it exists. It buys real time and it is safe.

Hold facts until the reasoning behind them is settled. Amounts, dates, names, quantities, yes-or-no answers about availability. These are the tokens that get revised, and they are also the tokens a caller acts on. Buffering a clause is worth the tens of milliseconds it costs.

Commit at boundaries where meaning is stable, not on a fixed token count. A clause or a sentence is a unit whose meaning will not invert; half a noun phrase is not.

The tension is real and does not resolve cleanly. Buffering more is safer and slower. The right balance depends on stakes: an agent reading back a delivery date should buffer more than one making small talk, because the cost of an unsayable correction scales with how much the caller cares about the fact.

Text to speech has its own minimum

One more constraint pushes in the same direction, and it is easy to miss when optimising for latency.

Synthesis needs context to sound right. Prosody (where stress lands, how pitch moves, whether the sentence sounds like a question) is a property of the whole phrase, not of individual words. Feed a synthesiser one word at a time and it cannot know where the sentence is going, so it produces flat, oddly-stressed speech that sounds robotic in exactly the way modern TTS is supposed to have solved.

So there is a floor on chunk size for natural-sounding output, and it fights directly against the floor on latency. Clause boundaries are the usual compromise: long enough for sane prosody, short enough to keep time-to-first-audio down.

Retrieved data makes this worse, because retrieved data is full of the things that need normalising before they can be spoken. $1,234.50 has to become "one thousand two hundred thirty four dollars and fifty cents". 2026-08-14 has to become a spoken date in the right convention. SKU-4471-B has to be read in a way a human can follow. None of that can be done on a partial token; you need the whole value before you can decide how to say it, which means these are natural buffer points whether you wanted them or not.

What "state" means while you are waiting

The word covers several things that drift apart during a retrieval turn, and conflating them is where the subtle bugs live.

At any moment mid-turn, the agent has:

• What was retrieved: the raw result, possibly partial, possibly stale
• What has been generated: model output, including tokens not yet spoken
• What has been sent to synthesis: committed, in flight
• What has actually been played: the only one the caller knows about

Under normal conditions these converge and nobody notices. Under a slow lookup, a timeout, or an interruption, they do not, and code that assumes the transcript equals the conversation will be wrong about what the caller knows. That divergence is the whole subject of part six, and a retrieval turn is where it starts.

Memory has its own version of this. Retrieved data has to enter context so later turns can use it, and dumping raw payloads is the obvious approach and the wrong one: it inflates every subsequent request, adds latency to turns that had none, and buries the two fields that mattered in a hundred that did not. Structure at ingestion: pull out what the conversation needs, keep a reference for the rest.

And retrieved data goes stale. A balance fetched at turn three may be wrong by turn nine, particularly if the agent itself has since done something that changed it. Deciding what to re-fetch, and when, is cache invalidation with a person waiting on the line. The pragmatic rule: anything the agent is about to act on, or state as fact, should be fresh; anything used for context can be as old as the call.

When it fails, and it will

Timeouts should be set against caller patience, not the dependency's SLA. If an internal service has a p95 of eight seconds, that is not a number to wait out on a phone call; it is a number to fail fast against. A caller will not wait eight seconds, so waiting eight seconds only converts a fast failure into a slow one.

Partial results are usually worth speaking. "I can see the order, but the tracking hasn't come through yet" is a better turn than either silence or a flat failure, and it is honest about which part worked.

Say what happened. "I'm having trouble reaching that system" is unsatisfying and it is enormously better than a pause followed by a change of subject. Callers forgive systems that admit limits; they do not forgive systems that appear to be ignoring them.

Never let a failed lookup produce a confident answer. This is the single most dangerous failure on this list, because it is silent. A model that receives an empty result and generates a plausible balance anyway has done something far worse than timing out, and on a phone call there is nothing for the caller to check it against. That is part seven, and the retrieval turn is where the exposure is highest.

The short version

A retrieval turn is the one place a voice agent cannot hide behind streaming, and it deserves specific design rather than the same path as every other turn:

• Start the lookup as early as intent allows, for idempotent reads
• Speak before the wait, conditionally, and never promise an outcome you do not have
• Stream framing, buffer facts, commit at boundaries where meaning is stable
• Normalise retrieved values before synthesis, and accept the buffering that implies
• Keep generated, spoken and heard as separate ideas, because they diverge here first
• Time out against the caller, not the API
• Fail out loud

Next in this series: barge-in. How to tell a caller genuinely interrupting from a cough, a backchannel "mm-hmm", someone else in the room, and your own agent's voice echoing back down the line.