What Breaks at 2 AM: Monitoring AI Voice Agents in Production

Your AI voice agent went live in March. It handled 200 calls on day one. By June, it is handling 4,000, and the team is asking why conversion is down 8%.

The answer is almost never the model. The answer is drift.

We learned this the hard way. New caller patterns, fresh objections, seasonal inventory questions, a script change someone made in the admin panel three weeks ago. None of it dramatic on its own. All of it accumulates until the agent that worked in Q1 sounds confused in Q2.

This post is about the part of voice AI that nobody puts in the demo reel: the boring, ongoing work of watching a production agent stay sharp. We will cover what we measure, what we ignore, and the four alerts that actually wake somebody up at 2 AM.

Why "deploy and walk away" stopped working

Three years ago, you could ship a voice agent, check the dashboard once a week, and call it good. The agents were simple. The scripts were simple. The caller population was narrow.

That era is over. The agents we ship in 2026 make hundreds of small decisions per call. The scripts adapt. The caller population is everyone who picks up a phone. A configuration change someone made to handle a Memorial Day promotion in May can quietly degrade the agent's behavior in October if nobody notices.

The model is the same. The world around it moved on.

What drift actually looks like

Drift is not a single thing. It is a cluster of small changes, and the only way to catch them is to measure them.

Intent drift. A cluster of calls that used to be "test drive booking" slowly becomes "test drive booking OR general pricing question OR warranty follow-up." The agent keeps responding as if it is the first one, and conversion in that cluster drops. The intent definition was correct in March. It is not correct anymore.

Escalation drift. The escalation rate was 11 percent in Q1. In Q2 it is 14 percent. On the surface, that is a 3 percent change. In practice, it means 90 extra calls per week are being kicked to a human for reasons the dashboard cannot tell you. Some of those reasons are good. Some are not.

Sentiment drift. Caller sentiment scores in the "service appointment" cluster go from 0.62 to 0.54 over six weeks. Nobody on the team can hear a difference when they sample calls. The aggregate sees it. The samples do not.

Script drift. A new car model launches. The dealer adds a line to the script. The line is great for that model and quietly confusing for the other seven the dealer carries. The overall conversion looks fine. The conversion for the other seven models is in free fall.

All four are drift. None of them are model problems. All of them are invisible to a single-call sample.

The four numbers that actually matter

Most voice AI dashboards show twenty metrics. We watch four.

Conversion rate per intent cluster. Not the headline conversion rate. The cluster-by-cluster rate, broken out weekly. The headline number can hold steady for six weeks while three clusters are silently bleeding.

Escalation rate by reason code. Not the overall escalation rate. The rate per reason, with the reasons tracked over time. A "user request" escalation that is 60 percent of all escalations in March should not be 60 percent in June. If it is, something changed.

Average handle time per cluster. A 20-second bump in handle time on the "price quote" cluster means the agent is hesitating on something. It is almost always a script or a knowledge gap, not a model gap.

Sentiment score per cluster. The most underrated number in voice AI. Sentiment drift is the earliest signal that the agent is starting to confuse or annoy the caller, and it usually shows up 3 to 6 weeks before conversion moves.

The alerts that actually fire

We have a dozen alerts wired up. Most of them are noise. Four are not.

Alert 1: cluster conversion drops more than 5 percent week over week. This is the canary. It almost always points to a real change, either in the caller population or in the agent's config. We page on this.

Alert 2: "user request" escalations exceed 20 percent of all escalations for two consecutive days. When the agent is escalating because the user asked for a human, that is expected. When it is escalating because the agent is confused, the user is asking for a human, and the percentage of those escalations is climbing, that is a problem. Two days is the threshold to filter out lunch-rush noise.

Alert 3: handle time on a single cluster exceeds the 90th percentile of the trailing 60 days. This catches the slow-burn knowledge gap. The agent is taking longer to answer something it used to answer quickly. The caller is patient, but not infinitely patient.

Alert 4: sentiment drift in any cluster exceeds 0.08 over a 14-day window. Small per-call. Loud in aggregate. We do not page on sentiment, but we surface it in the Monday review because it is the most reliable early signal of the four.

What we ignore

Plenty of signals are noise. The model perplexity on any single call. The exact transcription of any individual utterance. The number of times the agent said "um" in a given shift. Per-call metrics are debugging tools, not monitoring tools.

The temptation is to wire up alerts on everything. Resist it. The team that pages on twenty alerts is the team that ignores alerts. We wire up alerts on the four numbers above, and we let the rest live in the dashboard for human review.

What we do weekly

Every Monday, the engineering lead spends 45 minutes on a single review.

The review is the same every week. Pull the four numbers. Plot the four numbers. Look for the cluster where at least one of the four moved by more than the threshold. Open the calls in that cluster. Read the transcripts. Identify the change.

Most weeks, the change is small. A new objection pattern. A model of car the dealer just added to inventory. A pricing update that did not make it into the agent's knowledge base. We make a small fix on Tuesday. We re-measure on Friday.

Some weeks, the change is bigger. A new competitor running a promotion the agent does not know about. A recall that is driving 40 percent of the call volume in a region. A weather event. We escalate to the dealer. We re-cluster intents. We re-evaluate escalation thresholds.

Almost never, the change is the model. When it is, the fix is also boring: a config change, a prompt update, a small addition to the knowledge base. We do not retrain.

The lesson

Production voice AI is not "deploy and walk away." It is closer to running a service. The agents that work in 2026 are the ones whose builders are watching them.

The monitoring is not glamorous. The dashboard is not exciting. The Monday review is forty-five minutes nobody wants to do. We do it anyway. It is the difference between an agent that works in Q1 and an agent that still works in Q4.