Sub-200ms barge-in: the latency budget behind natural-sounding voice AI

Most voice agents feel slow. The reason is almost never the model. The reason is barge-in.

Barge-in is the moment a caller interrupts the agent mid-sentence, and the agent stops talking inside a few hundred milliseconds instead of finishing its scripted line. Every voice platform on the market claims to support it. Almost none of them do it in under 200 milliseconds. The ones that claim to be doing it under 200 are usually measuring the wrong thing.

We spent most of this quarter trying to make ours do it for real. Here is what the latency budget actually looks like, what we had to give up to hit it, and the metric that turned out to matter more than the one we were originally chasing.

What we were measuring first

Our first dashboard tracked end-of-turn detection latency, which is the time between the caller finishing their sentence and the agent starting to respond. That is the number most voice platforms publish. It is a fine number. It is also the wrong one for natural-sounding conversation.

End-of-turn detection is what happens when the caller is done. Barge-in is what happens when the caller is not done. A voice agent can have a 400 millisecond end-of-turn response and still feel broken, because the caller spent the last 1.2 seconds of their own turn yelling over the agent's last sentence.

The number that actually matters is the time between the caller's voice entering our system and the agent's audio stopping. We call that interruption latency. It is harder to measure, harder to hit, and the only one a caller can feel.

The 200 millisecond budget

Once we started measuring interruption latency, the budget fell out fast. A human interrupting another human on a phone call lands somewhere between 150 and 250 milliseconds. We picked 200 as the target. That is faster than the slowest human, slower than the fastest human, and the point at which callers stop hearing a delay at all in our testing.

Splitting 200 milliseconds across the pipeline is where it gets interesting.

Speech-to-text first pass: 30 to 60 milliseconds. Streaming STT gives us a partial hypothesis as the caller is talking. We act on the partial, not the final. The 30 to 60 ms is the time to first token of the partial, not to a finalized transcript.

End-of-turn classification: 20 to 50 milliseconds. This is the model that decides whether the caller is about to keep talking or has paused. We use a small classifier here, not the main agent, because the main agent is too slow and too cautious.

Interrupt decision and audio cut: 20 to 40 milliseconds. Once we decide to interrupt, we have to actually stop the TTS audio. Most TTS engines buffer. The buffer is the killer. We run our TTS with a 40 millisecond buffer cap and a hard cut on interrupt signal.

Network and codec overhead: 30 to 80 milliseconds. This one we cannot control. WebRTC adds latency. Cellular networks add latency. The dealer phone system adds latency. We budget 80 ms in the worst case and 30 in the best case.

Add it up: best case 100 ms, worst case 230 ms. The 200 target is real but it is not safe. We hit it on roughly 70 percent of interruptions in production, and we hit sub-300 on the rest. The 30 percent that miss are the ones callers notice.

What we had to give up

The first thing we gave up was waiting for a finalized transcript. Acting on partials means acting on words the caller may not have finished. The first month of rollouts had a small bump in false interrupts, mostly on common filler words like "yeah" and "ok" that the caller was using as acknowledgment, not as a real turn.

We tuned the classifier to ignore short acknowledgments and only trigger on partials that look like the start of a new clause. False interrupts dropped by half. We are still paying a small tax on it. Some real interruptions get a 50 to 100 millisecond delay while the classifier waits to be sure.

The second thing we gave up was TTS quality. The fastest TTS engines do not have the best voice quality. We picked the best-sounding engine we could find that still let us cut the buffer in under 40 milliseconds. That meant giving up two candidate engines that had warmer voices but buffered 120 ms ahead.

The third thing we gave up was the easy debugging story. End-to-end interruption latency is hard to instrument. We had to build a custom trace that follows a single interruption event through STT, the classifier, the interrupt signal, and the TTS cut. Each one is fast. Together they are fragile. A garbage collection pause on the agent side can blow the budget for a single call. We added jitter buffers and accepted a small hit to the average to keep the tail under control.

The metric that mattered

We were originally chasing average interruption latency. The 200 millisecond target was an average target. The first week of data looked fine. The p95 was not.

Average interruption latency is a vanity metric. Callers do not experience the average. They experience the worst call of the day, and the worst call of the day is the one where the dealer calls back. We started tracking p95 interruption latency, then p99, and we made those the targets.

p95 is currently sitting around 280 ms. p99 is around 410 ms. The 410 ms outliers are what callers remember. The next quarter of work is almost entirely about that p99, not about moving the average down by 20 ms.

What this means for dealers

If you are evaluating a voice AI vendor, ask them for interruption latency, not end-of-turn latency. Ask for the p95, not the average. Ask them to show you a recording where the agent gets interrupted mid-sentence and stops within 200 milliseconds. If they cannot show you one, they do not have the metric.

A voice agent that handles barge-in well sounds like a person who is listening. A voice agent that does not handle it well sounds like a person who is waiting for their turn to talk. Callers can tell the difference inside the first minute, and the call goes one of two ways from there.

Frequently asked questions

What is barge-in for a voice AI agent? Barge-in is the ability for the caller to interrupt the agent mid-sentence. Real barge-in stops the agent's audio within roughly 200 milliseconds of the caller speaking, rather than waiting for the agent to finish its scripted line.

Why is barge-in latency the most expensive feature in a voice agent? It sits on the critical path of every turn. Streaming speech-to-text, end-of-turn detection, agent decision-making, and text-to-speech all have to react inside the same 200 ms window, with no room to buffer.

What is a realistic barge-in latency target for production voice AI? Under 200 milliseconds end-to-end is the bar for natural conversation. 300 to 400 milliseconds is usable but starts to feel like a polite assistant. Anything over 500 milliseconds is the uncanny valley where callers start talking over the agent on purpose.

Does faster barge-in hurt transcription accuracy? Often yes. Partial hypotheses are noisier than finalized transcripts, and acting on a partial hypothesis means acting on words the caller may not have finished. The trade is a small bump in false interrupts in exchange for not making the caller wait.