Voice agents your callers mistake for your best rep

Human turn-taking runs on a rhythm of roughly 200 milliseconds between speakers. Telephone conversation tolerates more, but not much: past about a second of silence, callers assume the line has dropped or that they were misunderstood, and they start repeating themselves — which produces overlapping speech, which produces further confusion. Nearly every "the voice bot felt robotic" complaint traces back to timing rather than voice quality.

Meeting the budget requires streaming at every stage. Speech-to-text emits partial transcripts as the caller speaks rather than waiting for the end of the utterance. The language model streams tokens, and text-to-speech begins synthesising the first phrase while the rest is still being generated. Done well, the agent starts speaking before it has finished deciding what to say — exactly as people do.

It also requires ruthless discipline about in-turn work. A CRM lookup that takes 800 milliseconds consumes the entire budget on its own. We solve this with speculative fetching — starting the likely lookups as soon as intent is recognised, before the caller has finished — and with a spoken acknowledgement covering genuinely slow operations, because "let me pull that up" is natural human behaviour and buys a second of silence legitimately.

The last piece is endpointing. Deciding when the caller has finished is a judgement call, not a timer: people pause mid-sentence to think, and different populations pause for very different lengths. We tune endpointing against your real call recordings and use semantic signals — whether the utterance is grammatically complete — alongside acoustic silence.

Voice agents are excellent at bounded, transactional conversations with a clear objective: booking, status, verification, routing, reminders, straightforward qualification. They are poor at open-ended advisory conversations, emotionally difficult calls, and anything requiring negotiation or discretion. Deploying them into the second category damages the brand and the programme.

We scope deployments intent by intent and put explicit escalation triggers on everything else. Distress in the caller’s voice, an explicit request for a person, three failed clarification attempts, or any topic outside the configured intents all route to a human immediately. The agent never argues, never insists it can help, and never loops.

There is also a category we advise clients to avoid entirely regardless of technical feasibility: collections calls to individuals, medical triage without clinical sign-off, and any outbound campaign where the recipient has not consented to contact. The technology can do these things; the regulatory and reputational exposure is not worth it, and we will say so before you ask.

Voice is the most regulated AI channel we work in, and the rules differ by jurisdiction. Several US states now require disclosure that the caller is speaking to an AI system, the EU AI Act requires it for interactions with natural persons, and India’s telecom regulations govern outbound calling independently of AI. Our default is disclosure on every call in the opening line, regardless of local requirement — it costs nothing and it removes the entire category of "we were misled" complaints.

Recording consent is handled per jurisdiction and captured verifiably at the start of the call, with the consent state stored alongside the recording. Where two-party consent applies, the agent obtains it explicitly before recording begins rather than assuming it from a notice.

Payment handling is where most voice deployments quietly create risk. We do not have the agent collect card numbers by voice: it sends a secure payment link by SMS mid-call, or transfers to a PCI-compliant IVR for the payment portion and resumes afterwards. This keeps card data entirely outside the transcript, the recording and the model context.

Retention is configurable and defaulted conservatively: transcripts and recordings held for the period your policy requires, redaction applied to sensitive fields before storage, and deletion honoured on request. For healthcare deployments we operate under BAA-equivalent terms with all processors in the chain.

Replacing your telephony to pilot a voice agent is unnecessary and we do not recommend it. Voice agents sit alongside existing infrastructure: a SIP trunk, a Twilio number, or a queue in your contact-centre platform routes a defined slice of traffic to the agent, and everything else flows as before.

The safest first slice is calls that currently go unanswered — after-hours, holidays, and overflow past a queue threshold. There is no comparison case where a human would have done better, so the pilot can only improve the caller’s experience. Once containment and QA scores are proven on that slice, we expand into business-hours traffic for specific intents.

Warm transfer is the integration detail that most affects perceived quality. When the agent hands off, the receiving human should get a spoken or on-screen summary of what has already happened, plus the caller record open. Cold-transferring a caller who then has to repeat everything undoes the benefit entirely, and it is the most common integration shortcut we are asked to fix.

A traditional QA team samples perhaps 2% of calls. A voice agent deployment can score all of them, automatically, on the same rubric — task completion, script adherence, disclosure delivered, escalation appropriateness, interruption handling, and caller sentiment trajectory.

That coverage changes what is knowable. Instead of anecdotes about problem calls, you get distributions: which intent has the worst completion rate, which time of day produces the most escalations, whether accent robustness varies by region, and how each is trending week over week. Failures become a prioritised backlog rather than a debate.

We apply the same scoring to your human agents where clients want it, which is often uncomfortable and always informative — in several deployments the AI agent’s script adherence and disclosure consistency exceeded the human baseline within the first month, while human agents remained clearly better at the difficult conversations. That is the right division of labour, and the data makes the case for it.