What's the realistic streaming STT latency target in 2026?

Under 200ms median first-token latency on a tuned pipeline. The platform-level providers (Deepgram, AssemblyAI, the hyperscalers) all hit this in production for English. The remaining variance is in the connector layer and the network path between provider and inference. If you're seeing 400ms-plus, it's almost never the STT model.

If STT isn't the bottleneck, what is?

Endpointing, turn-taking, and barge-in handling. The model can transcribe in real time, but the agent still needs to know when the caller is done talking, when they've interrupted, and when a pause is thinking rather than finished. That's where most production voice systems still lose milliseconds and, more importantly, lose the conversational rhythm.

Why does endpointing matter more than raw STT speed?

Because callers feel turn-taking. A 100ms gap at the right moment sounds natural. A 200ms gap at the wrong moment sounds like the agent ignored them. Optimizing the model when your real loss is in the conversational handoff is engineering in the wrong layer.

What's the practical change teams should make in 2026?

Stop measuring end-to-end call latency as the headline metric. Split it: STT median, endpointing decision latency, LLM first-token, TTS first-audio. Most of the time the metric that looks bad is in the agent orchestration layer, not the speech layer.

AI Technology · 2026-06-21

Streaming STT is no longer the bottleneck: what changed in 2026

Streaming speech-to-text latency dropped below 200ms median in production by mid-2026. The conversation in voice AI has moved upstream to endpointing, turn-taking, and what to do when the caller is mid-thought. Here's what the new architecture looks like.

For most of the last three years, the voice AI conversation in production has been about latency. Get the first token out fast. Beat the 500ms barrier. The whole architecture optimized around shaving milliseconds off the speech-to-text pipeline because that was the slowest leg.

In 2026, that conversation has changed. Streaming STT is no longer the bottleneck.

What actually changed in 2026

Three things converged.

First, the platform-level STT providers all shipped streaming models that hit sub-200ms median first-token latency on production workloads. Deepgram, AssemblyAI, the hyperscalers. They each got there a different way, but they all got there. The 400ms-plus numbers that defined production voice AI in 2023 are now an outlier signal, not a default.

Second, the connector layer between provider and inference got more attention. Network paths, regional endpoints, gRPC versus WebSocket, batching tradeoffs. These used to be "engineering hygiene" concerns that didn't move headline numbers. In a 200ms world, they do. A poorly tuned connector can add 80ms before the model even sees the audio.

Third, the model providers started publishing per-region and per-language latency distributions instead of just aggregate numbers. That data exposed where the real variance was hiding. For most production teams, it wasn't where they thought.

The result is that streaming STT is now table stakes. A team that can't hit sub-300ms end-to-end STT in 2026 isn't behind on a research problem. They're behind on an integration problem. Those have different fixes.

The new bottleneck: endpointing and turn-taking

If the model is fast, what isn't?

Turn-taking. Endpointing. Barge-in handling. The agent needs to know when the caller has finished talking. When they've paused to think versus paused to end. When they've started speaking over the agent. None of these are STT problems. They're orchestration problems, and they sit on top of the speech layer.

A 100ms gap at the right conversational moment sounds natural. A 200ms gap at the wrong moment sounds like the agent ignored them. The loss isn't in the speed. It's in the timing. The same millisecond budget can produce a great conversation or a terrible one depending on where it lands in the rhythm of the call.

This is where the interesting engineering work in 2026 has moved. Endpointing models that decide in 50-150ms whether a pause is a turn boundary or a breath. Barge-in detectors that distinguish a caller interrupting intentionally from a background noise spike. LLM orchestration layers that can start generating while endpointing is still deciding, and back out cleanly if the caller keeps talking.

The architecture looks different than the 2023 version. The pipeline is more parallel, less linear. The agent doesn't wait for the STT segment to complete before starting work on the response. It commits to a draft, holds it tentatively, and either confirms or replaces it based on what the caller does next.

How to measure the right things

The mistake most teams are still making in 2026 is reporting end-to-end call latency as the headline metric. It's the number leadership asks for, it's the number vendors quote, and it's almost useless for diagnosing production issues.

The metrics that actually matter split the pipeline:

STT median first-token: the time from end-of-speech to first transcribed token. Should be under 200ms in 2026.
Endpointing decision latency: the time from end-of-audio to turn-finality signal. This is where the real variance lives.
LLM first-token: the time from prompt submission to first response token. Model-dependent, but usually 150-400ms.
TTS first-audio: the time from first response token to first audio frame out. Streaming TTS can hit sub-100ms.

Add them up and you get the total latency budget. But the breakdown tells you which leg to optimize. Most teams in 2026 are surprised to find the bottleneck isn't where they assumed.

What this means for production architecture

The teams running voice AI at scale in 2026 have stopped treating the pipeline as a sequential chain. They're running STT, endpointing, LLM, and TTS in parallel where possible, with the agent orchestration layer holding tentative responses until the turn-finality signal confirms them.

It's not free. The orchestration layer needs to handle backout cleanly when a caller keeps talking past the agent's first draft. The endpointing model needs to be accurate enough that the backout rate is low. The LLM needs to support partial generation without committing until the orchestration layer says so.

But the user experience is markedly better. The conversation has rhythm. The agent doesn't talk over the caller. The pauses feel intentional rather than mechanical.

The 2023 voice AI pitch was "we hit 800ms latency." The 2026 pitch is "we hit sub-200ms STT, our endpointing decision lands in 80ms, and our orchestration layer holds tentative responses so the conversation has flow." The buyers who understand the difference win on user experience. The buyers who still lead with end-to-end latency numbers are buying a metric that stopped mattering eighteen months ago.

The teams shipping production voice AI right now are making architectural bets on endpointing, orchestration, and turn-taking rather than on the STT model itself. If you're evaluating a voice AI platform in 2026, the right questions aren't about model latency. They're about how the platform handles the moments between the tokens.