DTMF Voice AI Needs a Separate Event Path

A DTMF voice AI system should treat each keypad press as a structured telephony event. It should not pretend the digit came from speech recognition, splice it into the transcript, or let the language model guess what the caller meant.

That separation sounds small. It affects turn state, interruption handling, tool safety, and the audit trail. A caller pressing 1 may be choosing an option, confirming a prompt, or entering the first digit of a longer value. The keypress is certain. Its meaning depends on the active prompt and the application state.

Twilio's Media Streams WebSocket documentation describes DTMF as its own message type: "A dtmf message is sent when someone presses a touch-tone number key in the inbound stream, typically in response to a prompt in the outbound stream." That event boundary should survive all the way through the voice agent.

Keep DTMF out of speech

Speech and keypad input can reach the same conversation, but they are different evidence. Speech recognition returns a transcript, timing, and sometimes a confidence value. A keypad event returns a digit selected by the caller. Converting both into a loose text string throws away the distinction.

A cleaner event model keeps the source visible:

{
  "type": "dtmf",
  "digit": "1",
  "sequenceNumber": "5",
  "callId": "current-call",
  "promptId": "department-choice"
}

The first three fields mirror the shape documented by Twilio Media Streams. The application adds its own call and prompt binding so the digit can be interpreted in context. A 1 during a department menu is not the same action as a 1 during appointment confirmation.

Once the application validates the event, it may add a readable conversation entry such as "Caller selected sales." That entry is useful for the model and the audit record. It should still retain source=dtmf rather than masquerading as a spoken sentence.

Why transcript-only handling breaks

The language model should not decide whether an incoming digit belongs to the current prompt. That is application state.

Suppose the agent asks the caller to press 1 to confirm or 2 to change an option. The caller presses 1 while the agent is still speaking. If the telephony event is flattened into the transcript, several things can go wrong:

• The digit may appear beside a partial speech transcript and look like one utterance.
• A late event may be applied after the prompt has already timed out.
• A repeated event may trigger the same tool twice.
• The model may interpret a valid key outside the allowed options.
• The system may lose whether the input came from the keypad or speech.

These are state and idempotency failures. Better prompting cannot reliably repair them after the source context has been discarded.

The same principle applies to storage. Keep the raw event and its ordering metadata in the trace. Store the validated selection separately. A transcript can display the human-readable result, but it should not be the only record of what happened.

Give every prompt an input contract

Each prompt that accepts DTMF needs a small contract before the agent starts speaking. Define the allowed digits, the number of digits expected, the completion key if one exists, the timeout behavior, and the action that follows validation.

Twilio's <Gather> reference supports speech, DTMF, or both. It also makes the mixed-input rule explicit: "When you set this attribute value to dtmf speech, Twilio gives precedence to the first input it detects." If speech arrives first, the keypad path should not remain quietly open as a second chance to trigger the same decision.

A practical prompt state machine might use these states:

1. collecting: accept events bound to the current prompt.
2. complete: the expected input or completion key has arrived.
3. validating: confirm the digit count, allowed values, and prompt identity.
4. accepted: create one conversation event and authorize the next read or action.
5. rejected: explain the allowed choices without changing external state.
6. expired: ignore late input and move to the defined fallback.

The names do not matter. The transition rules do. Only one state should authorize an external action, and replaying the same event should not authorize it again.

Decide how keypad input affects playback

A caller may press a key before a prompt finishes. That can be useful. It can also leave the agent's audio playing after the selection has already been accepted.

Twilio ConversationRelay keeps DTMF separate from speech at the WebSocket boundary. Its documentation says: "Conversation Relay sends this message when you turn on DTMF detection and the caller presses a key." The same reference allows text playback to be marked interruptible by caller speech or DTMF digits.

The application still needs a policy. For a simple menu, a valid key can stop playback and advance immediately. For a longer numeric entry, the first digit may stop the prompt but keep the collection window open. An invalid key might stop playback and trigger a short correction. None of those behaviors should be an accidental side effect of whichever callback runs first.

Track what audio was active when the digit arrived. If playback is cleared, record that transition. This makes it possible to explain why a prompt stopped and whether the caller heard the confirmation that followed.

Separate inbound keys from outbound tones

Receiving a caller's keypress and sending tones into another phone system are different operations.

Twilio Media Streams documents inbound DTMF events for bidirectional streams. Its Media Streams overview says outbound DTMF from the media server toward Twilio is not supported. ConversationRelay has a separate sendDigits message for sending digits to the call. The transport contract determines which capability exists.

Do not hide that distinction behind one generic dtmf() tool. Use names that state direction and effect, such as collect_keypad_input and send_digits_to_call. Apply separate authorization and verification rules. An inbound selection can often remain read-only until it is validated. Sending digits may control an external IVR and should be treated as an action.

DTMF is precise, not automatically safe

A keypress avoids speech-recognition uncertainty, but it does not make the surrounding workflow secure by default.

The system still has to bind the event to the correct call, current prompt, and allowed input set. It should avoid placing sensitive digit sequences in broad application logs, model context, analytics payloads, or ordinary transcripts. Workflows that collect payment or authentication data need purpose-built controls rather than a general conversation trace.

For ordinary menu choices, log only what the team needs to diagnose the flow: event source, prompt ID, accepted choice, transition, and resulting action status. Keep raw values only where the operating requirement supports them.

Test event order, not just the happy path

A test that presses 1 once and sees a log line proves very little. DTMF failures often appear when speech, playback, timeouts, and duplicate events overlap.

Run the same prompt through these cases:

• A valid key arrives while TTS is playing.
• Speech arrives first, followed by a key.
• A key arrives first, followed by speech.
• The caller presses the same key twice.
• An invalid key arrives before a valid one.
• A digit arrives after the prompt expires.
• The transport delivers a repeated event.
• The call disconnects in the middle of a multi-digit entry.

Assert the final prompt state, the readable conversation event, and the external action count. If the key confirms an appointment, for example, the test should prove that one appointment action was authorized and one confirmation path completed. A correct transcript is not enough.

Order should also be visible in the trace. Keep the transport sequence, application receive time, prompt state before and after the event, and any playback-clear or tool event that followed. Our guide to voice AI weekly review explains why audio, transcripts, and timelines need to be reviewed together. DTMF adds one more event lane to that packet.

Keep tool execution behind validation

The model may use the accepted selection to continue the conversation, but it should not receive a raw digit and immediately invent a tool call.

Put a deterministic validation step in front of tool authorization. It should answer four questions:

1. Does this event belong to the current call and prompt?
2. Is the prompt still accepting input?
3. Is the digit or completed sequence allowed?
4. Has this exact selection already been accepted?

Only then should the workflow create the normalized selection and allow the next action. The action keeps its own destination verification and idempotency controls. That boundary matches the pattern in our parallel voice AI tools guide: orchestration decides when enough valid evidence exists, then one current response or action proceeds.

The tool schema drift guide covers the next risk. If providers represent DTMF differently, normalize them at the adapter boundary and keep one stable application event schema.

A rollout that exposes mistakes early

Start with one low-risk menu whose accepted values only choose the next branch. Persist the raw telephony event, normalized selection, prompt transition, and response that followed. Review calls where input was rejected, repeated, or received after expiry.

Next, add a multi-digit flow without an external write. Test completion keys, timeouts, backspace or correction behavior if supported, and redaction. Only after the collection path is stable should a validated selection authorize an action outside the conversation.

When an action is added, verify the destination. A keypress can request a transfer or confirm a scheduled choice, but the system should report the outcome only after the downstream operation returns usable proof. If the outcome is unknown, keep the event and action state visible without replaying the selection.

Frequently asked questions

What is DTMF in voice AI?

DTMF is the signaling system behind telephone keypad presses. In a voice agent, the telephony layer should deliver each press as a structured event with a digit and call context. The application validates the event without asking speech recognition or a language model to infer what the caller pressed.

Should DTMF digits be added to a voice agent transcript?

Keep the raw event outside the speech transcript. You may add a separate conversation entry such as "Caller selected option 1" after validation, but preserve its source as keypad input. That keeps a verified keypress distinct from uncertain speech recognition.

Can a voice agent accept speech and keypad input at the same time?

Yes, but the application needs a clear arbitration rule. Twilio Gather gives precedence to the first input it detects when both speech and DTMF are enabled. Record which mode won, close the other path, and reject late events for the completed prompt.

How should teams test DTMF voice AI flows?

Test repeated keys, invalid digits, missing input, speech followed by a key, a key followed by speech, input during TTS, late events after timeout, and duplicate delivery. Verify the final state and any external action rather than stopping at the digit log.

A keypad press should stay a keypad press

DTMF gives a voice agent a precise input channel. The system wastes that precision when it turns the event into transcript text and asks the model to reconstruct missing state. Keep the raw keypress in its own event, bind it to the active call and prompt, validate it before changing state, and record what happened to playback. One accepted selection should authorize one current response or action.

That design is less magical than asking a model to sort it out. It is also much easier to test when a caller presses the wrong key, changes input modes, or hits the same button twice.