What sub-second latency changes about phone automation

We ran the same voice agent, on the same script, with the same voice, at two different latencies. At 1.8 seconds average response time, callers spoke slowly and loudly, used keyword-style phrasing — "APPOINTMENT. BOOKING." — and asked for a human 31% of the time. At 700 milliseconds, the same agent, the same script: callers spoke in full sentences, interrupted mid-answer the way people do, and asked for a human 9% of the time.

Nothing about the agent’s capability changed. What changed was the caller’s model of what they were talking to, and it changed at a threshold rather than gradually.

The cliff, and why it exists

Human turn-taking runs on a rhythm of roughly 200 milliseconds between speakers — remarkably consistent across languages and cultures. Telephone conversation tolerates more, because the channel already introduces delay and both parties expect it. But the tolerance has a limit, and it sits somewhere around a second.

Past that, a specific inference kicks in: the other party did not understand me. It is the correct inference in human conversation, where a long pause after a question usually means confusion. So the caller does what people do when they think they were not understood — they repeat themselves, louder and slower, often with different words. Which arrives as a second utterance while the agent is still processing the first, which produces genuine confusion, which produces another long pause.

The degradation is self-reinforcing, and it explains why latency problems in voice deployments never present as latency complaints. They present as "it does not understand people" or "it sounds robotic".

Where the budget goes

The response latency a caller experiences is the sum of four things, and it is worth knowing which one is eating yours.

System lookups are the component that most often blows the budget, and they are the one teams least expect, because they are not part of the "AI" at all. A CRM query that takes 800 milliseconds consumes the entire budget by itself. In one deployment we inherited, the agent was fast at everything except the availability check — which was a synchronous call to a practice-management system averaging 1.4 seconds. Every booking conversation had a two-second hole in it, and the client had spent two months evaluating alternative voice models.

Three techniques that buy the time back

Stream everything

Speech-to-text emits partial transcripts as the caller speaks rather than waiting for them to finish. The model streams tokens rather than returning a complete response. Text-to-speech begins synthesising the first phrase while the rest is still being generated. Done properly, the agent begins speaking before it has finished deciding what to say — which is exactly what people do.

Fetch speculatively

The moment intent is recognised — often halfway through the caller’s sentence — start the lookups that intent will need. If the caller says "I want to book", begin fetching availability immediately rather than waiting for them to specify a day. Most speculative fetches are useful; the wasted ones cost an API call, which is a good trade against a second of silence.

Acknowledge out loud

For operations that are genuinely slow and cannot be pre-fetched, say something. "Let me pull that up" buys a legitimate second, because it is what a human would do. The critical detail is that the acknowledgement must be immediate — an acknowledgement that itself arrives after a pause is worse than silence, because it confirms the caller’s suspicion that something is wrong.

Endpointing: the underrated problem

Before any of the above matters, the system has to decide the caller has finished speaking. Get this wrong and nothing else helps: too eager and the agent interrupts people mid-thought; too patient and every turn carries a half-second of dead air the caller experiences as hesitation.

A fixed silence threshold cannot solve this, because pause length varies enormously — by person, by emotional state, by what they are doing while they talk, and by what they are saying. Someone reading a policy number off a document pauses mid-sequence. Someone describing a problem pauses to think. Someone confirming a time does not pause at all.

The approach that works combines acoustic silence with a semantic judgement about whether the utterance so far is complete. "I would like to book an appointment for" is grammatically unfinished and should be waited on regardless of the pause length. "Tuesday afternoon works" is complete and can be acted on immediately. Modern endpointing models do this well; the tuning is in setting thresholds against recordings of your own callers, because a dental practice and a collections desk have entirely different rhythms.

The trade-offs you make to hit the budget

Latency is not free. Three things get traded away to reach it, and each should be a deliberate choice rather than an accident.

Model size. Routine turns — intent recognition, slot filling, confirmations — do not need a frontier model, and using one costs 200 to 400 milliseconds that the conversation cannot afford. We route the majority of turns to a small fast model and escalate only for genuine reasoning, which typically means the hardest few percent of a call.

Context length. Every additional token in the prompt costs time to process. Voice agents therefore carry a summarised conversation state rather than a full transcript, and retrieve knowledge selectively rather than loading a corpus. This is the opposite of the instinct in chat applications, where more context is generally better.

Deliberation. Chain-of-thought reasoning improves quality and destroys latency. In voice we do the thinking before the turn where possible — pre-computing likely branches while the caller is speaking — and accept a simpler in-turn decision. Where a genuinely complex judgement is needed, the correct design is to say so aloud and take the second, not to silently take it.

Interruption handling is the other half of feeling human

Latency gets the attention, but the second thing callers notice is what happens when they speak while the agent is speaking. People interrupt constantly in real conversation — to confirm, to correct, to cut short an answer they already have — and a system that plows on regardless feels like a recording no matter how good its voice is.

Proper barge-in means the agent stops speaking within about 150 milliseconds of detecting speech, discards the remainder of what it was going to say, and processes the interruption as the new turn. The technical trap is that text-to-speech has already been generated and often buffered ahead of playback, so "stopping" requires cancelling a stream rather than simply not sending more audio.

The harder problem is distinguishing an interruption from a backchannel. "Mm-hm", "right", "okay" are acknowledgements that mean keep going, and an agent that halts for every one of them is exhausting to talk to. We classify short utterances during agent speech and only treat substantive ones as interruptions — which sounds subtle and is the difference between an agent that feels attentive and one that feels nervous.

Noise makes this considerably harder. A caller on a street or in a warehouse produces plenty of audio that is not speech, and an agent that interprets it as barge-in stops mid-sentence for no reason. Voice-activity detection tuned against recordings from your actual caller environments is not optional; it is the difference between a system that works in the office and one that works in the field.

What we measure on every deployment

Four numbers, tracked continuously across every call rather than sampled.

• Response latency at p50 and p95, broken down by component, so a regression can be attributed to speech, model or a system lookup rather than debated.
• Interruption handling rate: how often the agent correctly stopped, and how often it stopped for a backchannel it should have ignored.
• Containment by intent, not overall — an aggregate figure hides the one intent that is failing.
• Transfer quality: did the receiving human get the summary and the record, and did the caller have to repeat themselves?

The p95 matters far more than the average here. A deployment averaging 600 milliseconds with a p95 of 2.4 seconds feels bad, because the slow turns are not evenly distributed — they cluster on the complex queries where the caller is already least patient. Optimising the mean while ignoring the tail is the most common measurement error we correct.

What the threshold changes about scope

This is the part that matters commercially. Above the latency cliff, callers behave as if they are using a menu system — short keyword utterances, no interruptions, low tolerance for clarification. That constrains you to menu-shaped tasks: routing, status lookup, simple confirmation.

Below it, callers converse. They give compound information in one sentence — "I need to move my Thursday appointment to sometime next week, preferably morning" — which a conversational agent can handle in one turn and a menu cannot handle at all. They correct themselves mid-sentence. They ask follow-up questions. The agent can ask a clarifying question without the caller assuming it has failed.

That expands the automatable set considerably: rescheduling with constraints, multi-part enquiries, verification flows with corrections, qualification conversations that branch. In the dental group deployment we ran, three intents covered 78% of call volume, and two of the three were only viable because the agent was fast enough for callers to speak naturally.

The corollary is that latency is not a polish item to address after launch. It determines what you can put in scope, so it belongs in the architecture from the first week — and a deployment that cannot hit the budget should reduce its scope rather than ship slow.