Why Your Voice Agent Sounds Robotic (It's Not the Model)

A voice agent that takes over a second to respond doesn't feel like a slow AI — it feels broken, in a way a slow web page never quite does, because a phone conversation has a real-time expectation baked into it that no one consciously notices until it's violated. Teams debugging this almost always start by looking at the language model. It's rarely where the time is going.

The latency budget, stage by stage

A typical cascaded voice pipeline spends roughly 100 to 300 milliseconds on speech-to-text, 350 to 1,000 milliseconds on the LLM turn, 90 to 200 milliseconds on text-to-speech, and another 50 to 200 milliseconds on network round trips between whichever vendors are stitched together — which adds up to somewhere between 600 milliseconds and 1.7 seconds before the user hears anything back. Humans expect a conversational turn in under 800 milliseconds; past about 1.5 seconds, people start to assume the call dropped or the system froze. The model inference is one piece of that budget, not the whole thing, and often not even the largest piece.

Where the real time actually goes

Most production voice-agent problems are architectural, not model-related — the delay compounds across the pipeline: streaming speech recognition, a decision layer, and speech synthesis, each adding its own hop, and if any of those legs are hitting a different vendor over the open internet instead of a co-located service, the network overhead alone can rival the model inference time. The fastest speech-recognition models tend to be the least accurate, and the most accurate tend to be the slowest — picking one without profiling where your specific traffic actually sits on that curve is a common way to inherit someone else's latency problem.

• Stream everything — partial transcripts into the decision layer, partial responses into speech synthesis — instead of waiting for each stage to fully complete.
• Co-locate the pipeline near the telephony edge; cross-vendor network hops are a bigger latency cost than most teams budget for.
• Profile speech-to-text accuracy against your actual audio conditions (accents, background noise, phone codecs), not a vendor's benchmark numbers.
• Use a smaller, well-grounded model for the conversational turn before reaching for a larger one — a fast, correct answer beats a slow, marginally better one on a phone call.

The 2026 target for a production voice agent sits around 600 milliseconds end-to-end. Hitting that isn't a model-selection problem — it's a systems problem, solved by streaming instead of waiting, and by treating every network hop between vendors as latency budget that has to be justified, the same way you'd treat it in any other real-time system.