Real-Time Voice AI Systems: Latency, Turn-Taking, and Trust

A voice agent that transcribes every word correctly can still feel broken. The transcript is not the conversation.

Watch someone talk to one of these systems and you will see the failure before you can name it. They stop mid-sentence because the agent started answering a question they had not finished asking. They wait through a pause long enough to feel like a dropped call. They repeat an order number three times, and the agent confidently reads back a different one. Nothing in the transcript is wrong. Everything about the interaction is.

That gap is the subject of this article. Real-time voice AI is not speech-to-text plus a language model plus text-to-speech. It is a control loop with a shared clock, and the hard part is not generating language. The hard part is deciding when to speak, when to stop, and when to hand the conversation to a human.

If you are building voice-enabled experiences, four constraints decide whether yours feels usable: turn boundaries, latency budget, entity fidelity, and escalation. They are not four separate features. They are four views of the same loop, and the sections below keep returning to one live exchange to show how they interact.

One Turn, End to End

Before the architecture debate, walk a single utterance through the system. A user says: "Check order three one—" and trails off.

Capture opens a stream. Partial transcription begins producing text before the utterance is complete. The turn detector watches for a boundary and has to decide whether the trailing dash is a pause or a completion. If it fires too early, the agent answers a question that was never finished. If it waits too long, the user hears dead air.

Suppose the detector fires. The model begins generating a response. The user, hearing the agent start, says "no, wait—" and barges in. Now the system must cancel in-flight inference, discard queued audio that has not played, and decide what it remembers about the turn it abandoned. If cancellation is slow, the agent keeps talking over the user. If cancellation is incomplete, the agent later acts on a sentence it never finished saying.

Assume the interruption is handled. The user repeats the order number. The agent captures a partial value, and now entity fidelity becomes the binding constraint: does it guess the missing digit, or ask for a digit-by-digit repeat? If it guesses wrong, it retrieves the wrong account and reports it confidently.

Finally, suppose the entity is ambiguous, the audio is unclear, or the request falls outside the agent's authority. Escalation is the last constraint in the loop, and it is a decision the system has to make explicitly rather than discover at runtime.

Every section that follows maps back to one of these moments. Turn boundaries govern the first decision. Latency budget governs how fast each transition happens. Entity fidelity governs what the agent is allowed to act on. Escalation governs when it stops acting at all.

Cascaded Pipelines vs Speech-to-Speech Models

Two architectures dominate, and they optimize for different things.

A cascaded design chains streaming automatic speech recognition (ASR) into a language model into streaming text-to-speech. Each hop adds latency, and each interface is lossy in a specific way: the moment audio becomes text, the system discards prosody, pacing, tone, and hesitation. That non-lexical channel carries real meaning. "Fine" delivered flat and "fine" delivered bright are different messages, and a transcript renders them identically.

Speech-to-speech models process audio directly and generate audio back. OpenAI's engineering writeup on its continuous voice work describes the shift plainly: earlier cascaded systems ran speech-to-text, the LLM, and text-to-speech in series, which added latency and ignored cues like tone and pacing. Training a model to natively understand and generate speech preserved details that transcription threw away and produced faster responses.

The catch is worth stating carefully, because it is where a lot of product assumptions go wrong. Even after speech-to-speech arrived, the same writeup notes the system still relied on a separate turn detector to decide when inference could begin. The model handled more of the interaction, but the interaction remained turn-based. Removing the serialization penalty is not the same as removing the need to decide when a turn is over.

That distinction drives the practical tradeoff. Cascading gives you inspectable intermediate text, easier logging, per-stage control, and cheap component substitution. Speech-to-speech gives you lower latency and preserved delivery cues, but a much harder debugging surface — when something goes wrong, you have less to look at.

My rule: choose cascading when you need auditability, per-stage tuning, or the ability to swap models without renegotiating the whole system. Choose speech-to-speech when conversational rhythm is the product. If you cannot articulate which of those you are buying, you are not choosing an architecture. You are choosing a demo.

Turn Detection Is the Hardest Part of the System

Human speakers hand off to each other in a fraction of a second, and listeners read that rhythm as competence. A voice agent inherits the same expectation and none of the same instincts.

The turn detector's job is unenviable. Guess too early and you cut the user off. Guess too late and the agent feels sluggish. Both errors are visible, and both get attributed to your product rather than to a classifier.

The naive signal — silence — is not the same as completion. Users pause mid-thought, breathe, and correct themselves. A detector that fires on the first quiet moment will interrupt people constantly. As a design heuristic, interruption tends to be the more damaging error: an agent that talks over someone reads as not listening, and in most conversational products users tolerate a short delay more readily than they tolerate being cut off.

That asymmetry is the whole tuning problem, but it is a heuristic, not a law. Bias toward waiting, but only up to the point where the delay itself becomes the complaint. Where that point sits is a product decision, not a model decision. In a workflow where a wrong cut-off costs a re-explanation, wait longer. In a workflow where the user is checking a status and wants speed, wait less. The threshold should be set by the cost of the error in your specific context, and you should be able to defend the number.

Semantic and prosodic cues help. Falling intonation, clause completion, and filler words that signal continuation all carry information a raw silence timer ignores. But no cue set is complete, because some users speak in long, unpunctuated stretches and never yield a clean boundary. Plan for that user. They are not an edge case; they are a Tuesday.

Interruption Handling and the Cost of Talking Over Someone

Barge-in — the user starting to speak while the agent is still talking — is usually implemented as "stop the speaker." That is the easy 20 percent.

A correct implementation is a state transition. It has to cancel in-flight inference, discard queued audio that has not played yet, and decide what the agent remembers about the turn it abandoned. Each of those is a separate decision with a separate failure mode.

The context problem is the one that bites later. If the agent keeps the interrupted sentence in its history, it will eventually act on something it never finished saying. The user hears a partial thought, interrupts, and ten seconds later the agent behaves as if the full sentence had been delivered. From the user's side, the agent is responding to a conversation that did not happen.

The mirror problem applies to the user's own speech. When someone is interrupted, their utterance may be a fragment. Treating that fragment as a complete instruction produces confident action on incomplete input.

Latency compounds here. A slow cancel path means the agent keeps talking after the user starts, which reads as not listening regardless of how good the eventual answer is. The cancel path is part of your latency budget, not a separate concern.

Treat interruption as a first-class state transition with an explicit policy: what is kept, what is dropped, what gets re-asked. And watch for the failure mode where the agent stops talking but still executes the tool call it had already started. Silence is not cancellation.

Building a Latency Budget You Can Defend

"Make it fast" is not a specification. Latency is a sum: capture, transport, ASR or audio encoding, inference, synthesis, and playback each consume part of the budget, and the user experiences the total.

Three different budgets matter, and conflating them hides your real problem. Time to first acknowledgment is how long before the user knows they were heard. Time to first audible token is how long before the agent starts speaking. Time to a complete, correct answer is how long before the task is done. A system can be excellent at the first two and terrible at the third, or the reverse. Instrument them separately.

Streaming versus buffered inference is where scaling behavior lives. Many production systems implement "streaming" as buffered inference over sliding windows, where each new window overlaps the previous one to preserve context. The transcripts are correct. The compute is not. The model repeatedly reprocesses audio it has already seen, and at low concurrency that waste is invisible.

At scale it becomes a cliff. NVIDIA's published work on cache-aware streaming ASR describes the mechanism: overlapping windows fill GPU memory with redundant activations and intermediate states, memory pressure forces slower execution or reduced batching efficiency, and latency begins to drift. Responses arrive later and later relative to spoken audio. The writeup is direct about the consequence — even small delays disrupt turn-taking and make interruption handling impossible.

That mechanism is the transferable part. The specific numbers in the same writeup — a median time to final transcription of 24 milliseconds independent of utterance length, a sub-900-millisecond end-to-end voice loop for local deployment, and stable latency at 127 simultaneous clients over a three-minute stream — are reported results under stated conditions. They are not a benchmark for your stack, and the deployment scope (local, component-specific) is easy to miss. Read them as evidence that the drift problem is solvable in a bounded configuration, not as a target you can assume.

The distinction that matters is drift. A system that is fast with one user and progressively slower with many is not a slow system. It is an unstable one, and unstable latency is worse than uniformly mediocre latency because you cannot design a turn policy around it.

Capacity is not GPU throughput. OpenAI's engineering account of rebuilding its voice infrastructure makes this concrete: voice sessions stay open and send frames continuously, so CPU-side stream handlers, queues, and network paths must scale alongside inference. Under real load, a supporting component saturated earlier than load-test estimates predicted, inference requests accumulated, and latency compounded. The capacity question changed from "how many requests can a GPU handle" to "how many concurrent sessions can the system sustain while keeping every frame on schedule."

That reframing is the one I would steal. Measure time-to-final transcription, time-to-first-audio, and p95 latency under concurrent sessions. Median latency on a quiet machine tells you almost nothing about whether your agent will feel responsive at 3 p.m. on a busy day.

Reasoning effort belongs in this budget, not in a separate reliability section. Deeper reasoning costs time, so the level should be set by what a wrong answer costs you. OpenAI's prompting guidance recommends starting at the lowest reasoning level that still gives the assistant enough intelligence for the workflow, then tuning up or down based on task complexity, latency tolerance, and failure cost. That is the right frame: reasoning effort is a latency dial, and it should be turned deliberately rather than left at a default.

Exact Entities Are Where Voice Agents Quietly Fail

In the interaction class this article targets — transactional voice workflows where the agent takes an action — the highest-value failures are usually entity-fidelity failures, not comprehension failures. That is a scoping judgment, not a universal hierarchy. In a different product, a comprehension error may be the one that matters most.

Voice workflows depend on exact values: order IDs, tracking numbers, confirmation codes, account numbers, phone numbers, email addresses. Speech makes all of them hard. Users speak quickly, group digits inconsistently, spell partial values, use filler, correct themselves mid-turn, and pronounce characters that sound alike. One wrong digit fails a lookup or retrieves the wrong account.

This is structural, not a model-quality problem. Homophone and near-homophone collisions in digits and letters exist in the signal itself.

The dangerous behavior is confident guessing. An agent that fills in a plausible digit produces a wrong lookup that looks like a correct answer, and the user has no reason to doubt it. OpenAI's realtime prompting guidance uses exactly this example: a user says "check order three one—" and gets cut off, and the bad response is "I'll check order 31 now." The good response is "I heard only part of the order number. Could you repeat it digit by digit?"

Two patterns follow. First, detect incomplete or ambiguous captures and ask for confirmation digit by digit rather than inferring. Second, read back critical values before acting, and keep the read-back cheap so it does not dominate the conversation.

The decision rule I use: any entity that triggers an irreversible action gets confirmed, even at the cost of an extra turn. In most workflows an extra turn is a small tax. A wrong account lookup is a trust event, and trust events are expensive to reverse.

What the Voice Channel Carries Beyond Words

Speech conveys information through delivery as well as wording. Tone, emphasis, hesitation, accent, and pace all carry meaning, and in some settings they carry the decisive meaning.

A recent research evaluation tested four production real-time voice systems — OpenAI's GPT Realtime 2, Google's Gemini 3.1 Flash Live, and Alibaba's Qwen3.5 Omni Plus and Omni Flash — on tasks where both the words and the delivery patterns conveyed meaningful information. The observed result: decisions rested largely on the words, as if the voice had been reduced to its transcript. The authors call this asymmetry the emotional intelligence gap of voice AI, and note that prompting systems to explicitly attend to vocal delivery improved performance only partially and inconsistently.

Keep the layers separate here. The observed result is that, on the tested tasks, these systems underused delivery cues. The article's inference is that the limitation may be architectural rather than instructional, because prompting helped only partially. The open question is whether that inference holds across other tasks, other model versions, and longer evaluation windows. One evaluation cannot settle a field-wide claim, and model behavior in this area is changing quickly.

The practical implication is still usable. Where delivery rather than wording carries the decision — the authors model emergency and security interactions — do not route that decision through a voice agent without a human check. That boundary is the current state of the evidence, not a permanent verdict. Broader cross-system and longitudinal results would be the evidence that changes it.

Escalation: Continue, Clarify, Retry, or Hand Off

Escalation is a feature you specify, not a fallback you hope for. The cleanest way to design it is as a single decision boundary with four outcomes: continue, clarify, retry, or hand off to a human.

Continue when the turn is complete, the entities are confirmed, and the request is inside the agent's authority. Clarify when the input is incomplete or ambiguous — a partial order number, an unclear word, a self-correction mid-turn. Retry when the same failure repeats, such as a second failed entity capture, because a third attempt rarely succeeds and usually erodes patience. Hand off when the request falls outside the agent's authority, when the user signals frustration, or when the decision depends on delivery cues the system cannot reliably read.

OpenAI's realtime guidance frames escalation the same way — define fast, reliable escalation and what the agent should say, with thresholds tuned to your use case. The thresholds are yours to set, but the four outcomes should be explicit in your system, not emergent from prompt wording.

What the agent says when it escalates matters as much as when it escalates. A vague handoff erodes trust faster than a clear refusal. "I can't help with that" is better than a confident wrong answer followed by silence.

Long sessions deserve a note here because they change the state the escalation logic reads. Context grows, and an agent that never forgets becomes slower and less predictable. If your escalation thresholds depend on conversation history, that history is now part of your reliability surface, not just your prompt.

What to Build First, and How to Judge It

Start with the narrow workflow where a wrong answer is cheap and the turn structure is simple. Do not begin with the highest-stakes use case, because you will spend your first month debugging turn boundaries instead of learning whether the product is useful.

Build the measurement harness before the polish. The point of logging is not to collect events; it is to make a pass/fail decision about each constraint. The table below is the one I would start with. It maps each constraint to a scenario, a failure signal, a trace to inspect, and the decision the result should trigger.

Constraint	Scenario	Failure signal	Trace to inspect	Decision
Turn boundaries	User pauses mid-thought	Agent answers an unfinished question	Turn-decision timestamp vs. utterance end	Bias the detector toward waiting; re-test
Barge-in cancellation	User interrupts mid-response	Agent keeps talking, or acts on an abandoned turn	Cancel latency; queued-audio depth; retained context	Fix cancel path before adding features
Latency drift	Many concurrent sessions	p95 latency grows over session duration	Time-to-first-audio and time-to-final under load	Treat as instability, not slowness
Entity fidelity	Spoken order number with a correction	Wrong value acted on without confirmation	Capture attempts; confirmation events; lookup result	Require digit-by-digit confirm for irreversible actions
Escalation	Unclear audio or out-of-scope request	Confident wrong answer, or silent failure	Escalation events; handoff outcomes	Tighten thresholds; add explicit handoff language

The numbers in that table are deliberately absent. Numeric thresholds should come from your own error costs, not from a generic benchmark, and inventing them here would be worse than leaving them to your context.

Test with adversarial speech: fast talkers, digit strings, self-corrections, background noise, and users who never pause cleanly. The demo voice is not your user.

If you already have evaluation discipline from multimodal system work — define the task, define the failure, then measure — reuse it here rather than judging by how the demo sounds. The same principle applies: a benchmark score is a result under agreed conditions, and reliability is what remains when ordinary inputs disturb those conditions.

The skills worth building next are streaming audio transport, state machines for turn and interruption handling, and evaluation design for interactive systems. Those three compound. Model capability will keep improving underneath them, and my working hypothesis is that the bottleneck keeps shifting from the model toward system scheduling, capacity planning, and escalation design. That is a watchpoint, not a forecast. If it holds, it is where the durable engineering work sits.

A voice agent is trustworthy when its turn boundaries, latency budget, entity handling, and escalation thresholds are explicit and measured — not when its demo sounds natural. Instrument time-to-first-audio and p95 latency under concurrent sessions first. Build the cheap, simple workflow first. And keep routing decisions that depend on how something was said through a human until the evidence changes.