Learn how AI voice agents work under the hood — ASR, NLU, LLM, TTS — and what separates a pilot from a production-ready system in contact centers, healthcare, and enterprise.
Published on May 6, 2026 | AI Voicebots & Voice Agents
US call abandonment rates hit 8.9% in 2024 — the highest figure recorded in thirteen years. The average live agent call costs $7.20. Self-service channels cost $1.84 per contact. Legacy IVR systems are failing callers and inflating costs at the same time.
The market has a clear answer: AI voice agents. But "AI voice agent" covers everything from a simple FAQ bot to a fully autonomous clinical intake system. Before you evaluate vendors or launch a pilot, you need to understand what's inside the box.
This guide explains the full pipeline in plain language — from the moment a caller speaks to the moment the agent responds. It covers the four core technologies, the production constraints that separate demos from deployable systems, and the metrics that matter when you're evaluating platforms.
What an AI Voice Agent Actually Does
An AI voice agent handles a caller's request through natural conversation instead of rigid menus. It listens, interprets intent, takes action, and responds — all in real time.
How it differs from legacy IVR and chatbots
Legacy IVR forces callers through fixed decision trees using keypad presses and keyword spotting. AI voice agents accept natural language, understand meaning, and handle phrasing the system has never seen before — without a rule update.
Text chatbots share some intelligence with voice agents but skip the hardest parts: converting audio to text accurately under noise and accents, detecting when a caller has genuinely finished their turn rather than simply paused mid-thought, and generating spoken responses with natural pacing and prosody.
The job it's designed to handle
AI voice agents handle structured interactions where callers need information, authentication, scheduling, or account changes. They resolve common requests without a live agent. When a request exceeds the agent's scope, it escalates with full conversation context — not a blind transfer that forces the caller to start over.
Where it fits in the stack
A voice agent sits between your telephony infrastructure and your backend systems. It intercepts calls before they reach human agents, resolves what it can, and passes the rest forward with context. Its job is to shift more calls into the self-service tier without degrading the caller experience.
The Four Technologies Inside Every Voice Agent
A production voice agent succeeds or fails on four components. If one breaks, the caller feels it immediately.
ASR: turning speech into text
Automatic Speech Recognition (ASR) converts the caller's audio into a text transcript. It is the pipeline entry point — every downstream component depends on its accuracy.
Modern ASR uses streaming architectures that produce partial transcriptions continuously. This lets the system begin reasoning before the caller finishes speaking, which is the primary driver of low end-to-end latency. Voice Activity Detection (VAD) runs alongside ASR to determine when audio contains speech, reducing unnecessary computation on silence and background noise.
NLU: understanding what the caller meant
Natural Language Understanding (NLU) takes the transcript and extracts four signals:
- Intent — what the caller wants to accomplish
- Entities — specific data: account numbers, dates, amounts
- Sentiment — emotional state: frustrated, confused, satisfied
- Context — how this utterance connects to prior turns
Older platforms required developers to predefine every intent manually. LLM-based NLU understands meaning rather than matching literal keywords, which means it handles novel phrasing without a rule update.
The decision engine: between listening and responding
The decision engine maintains conversational state across multiple turns. It determines the next action: look up an account, process a transaction, open a ticket, or escalate to a human agent.
In regulated environments, a useful separation keeps LLMs handling language understanding while deterministic flows handle action selection and policy enforcement. This gives full auditability and makes it harder for edge cases to trigger unauthorized actions.
TTS: generating a response the caller can hear
Text-to-Speech (TTS) converts the agent's text response into spoken audio. Production TTS starts generating audio before the full response text is ready — it streams partial output to reduce wait time.
Voice quality matters: callers hang up on robotic audio. Modern TTS engines produce speech with natural prosody, appropriate breathing, and emotional inflection. Streaming synthesis — where audio generation begins from the first tokens rather than waiting for the full sentence — is the standard for any deployment targeting sub-second response times.
How a Single Call Moves Through the Pipeline
The canonical pipeline looks like this:
Audio In → VAD → STT → LLM → TTS → Audio Out
Each stage is independently testable, swappable, and optimizable. That modularity is what makes the pattern so practical for production systems.
From first word to transcribed text
Audio packets arrive from the telephony layer (SIP or WebRTC). VAD detects speech onset and begins feeding audio to the ASR model. Streaming ASR produces interim transcriptions in milliseconds — partial results that let the LLM start reasoning before the caller has finished speaking. A turn-detection model then determines when the caller has genuinely completed their thought.
Typical STT latency: ~200 ms with a streaming model.
From text to intent to action
The transcript passes to the NLU layer, which classifies intent and extracts entities. The decision engine checks conversational state, applies business rules, and selects a response path. If a knowledge lookup is needed, Retrieval-Augmented Generation (RAG) adds relevant context. The LLM generates a text response, streaming tokens as they are produced.
LLM time-to-first-token: typically 300–800 ms depending on model size and infrastructure. This is the single largest contributor to total pipeline latency.
From text to spoken audio
Response text streams to TTS, which begins generating audio from the first tokens. TTS time-to-first-audio: 100–300 ms in streaming architectures. Audio then streams to the caller via WebRTC or your telephony trunk.
Naive vs. streaming: a 1.5-second difference
In its naive form, each stage waits for the previous one to complete — result: 1,500–2,000 ms of minimum delay. Unusable in production.
In its streaming form, stages overlap: STT sends partial transcripts to the LLM while the user is still speaking; the LLM sends its first tokens to TTS before it has finished generating; TTS starts synthesizing from the first words it receives. With this overlap, latency drops to 400–800 ms — enough for natural conversation.
| Approach | How it works | Typical total latency |
|---|---|---|
| Naive (blocking) | Each stage runs to completion before the next starts | 1,500–2,000 ms+ |
| Streaming | Stages overlap — partial STT feeds the LLM while TTS synthesizes in parallel | 400–800 ms |
Under 1 second, conversation feels natural. Over 2 seconds, something feels broken — even if the words are perfect.
For a deeper look at where latency accumulates and how to cut it, see our guide on voice bot latency: anatomy, sources, and optimization.
What Determines Whether a Voice Agent Works in Production
Production performance comes down to three variables: accuracy, latency, and domain fit. If any of these fail, the system stops feeling conversational and callers stop using it.
Accuracy under real-world noise and accent variation
Aggregate Word Error Rate (WER) tells only part of the story. A system scoring 17% WER on clean office audio can degrade to 68% WER on live telephone data with background noise — a 4x error rate increase. Accented speech compounds the problem further.
Practical accuracy targets:
- Below 10% WER: Excellent — suitable for most production use cases
- 10–20% WER: Acceptable with careful monitoring and fallback handling
- Above 20% WER: Functionally unusable — errors cascade through NLU into wrong intent classification
Every transcription error propagates downstream: a misheard word changes intent classification, which changes the agent's response, which frustrates the caller. Demand production benchmarks measured under realistic conditions, not clean-audio controlled recordings.
Latency across the full pipeline
| Stage | Target latency | Notes |
|---|---|---|
| Audio transport (WebRTC) | < 50 ms | Requires a global low-latency media network |
| VAD | 10–50 ms | Runs locally, minimal overhead |
| STT (first partial result) | 100–200 ms | Streaming STT required |
| LLM time-to-first-token | 300–800 ms | The dominant bottleneck |
| TTS time-to-first-audio | 100–300 ms | Streaming synthesis required |
| Total (perceived) | < 1 second | Target for natural conversation feel |
The biggest improvement lever isn't faster ASR. It's co-locating your inference infrastructure with your media infrastructure, and reducing LLM invocations through template responses for predictable queries.
Domain vocabulary and runtime customization
When callers use terminology the base model hasn't seen — product names, medical terms, financial identifiers — accuracy degrades silently. The system confidently misclassifies intent rather than flagging uncertainty. Runtime vocabulary tools like contextual biasing let you supply session-specific terms to the ASR decoder without retraining the full model.
Turn Detection: the Most Underestimated Part of the Design
Turn detection is how a voice agent decides when a user has finished speaking. Get it right and conversations feel natural. Get it wrong and users notice immediately.
A voice agent that cuts you off mid-sentence feels rude. One that waits three seconds after you stop speaking feels broken. The engineering behind that tiny window — the moment between when you finish a thought and when the agent starts responding — determines whether a conversation feels natural or painful.
VAD-only monitors incoming audio and triggers on silence above a threshold. A silence timeout of 800 ms adds nearly a full second to every response before the pipeline even starts.
STT endpointing uses an end-of-utterance signal from the STT provider — faster than waiting for silence, and the solid default for most production agents.
Model-based detection reads the partial transcript in real time and predicts turn completion based on semantic meaning. This handles cases like "I want to book a flight to... uh... New York" where VAD would prematurely trigger on the pause.
Barge-in — the ability for a caller to interrupt the agent mid-response — requires keeping turn detection active during audio playback and cancelling the TTS stream immediately when the user speaks. Without it, agents feel robotic and rigid.
Where Voice Agents Are Deployed and What They Replace
Contact center triage and FAQ deflection
Contact centers deploy voice agents to contain calls by resolving requests without a live agent. A Forrester study of a composite organization documented 28% contact containment and $8.8 million in savings over three years.
Common deployments: first-line triage that routes calls correctly without a menu, FAQ deflection for hours and policies, order status and account balance inquiries, callback scheduling and queue management.
Healthcare scheduling and clinical intake
Healthcare voice agents handle appointment scheduling, benefits verification, and clinical intake. HIPAA requires a Business Associate Agreement (BAA) with every component that touches protected health information — including the STT engine, LLM service, telephony platform, and EHR integration.
Pipeline architectures have a clear advantage here: text sits between every stage, which means you can enforce PII redaction at the STT output before data reaches your LLM, and you can audit exactly what was said, transcribed, and generated at every step.
Financial services authentication and account queries
PCI-DSS adds a specific constraint: if your AI infrastructure processes audio containing verbal card numbers or DTMF tones — even transiently — that infrastructure becomes part of the cardholder data environment. Architecture mitigations include DTMF masking and out-of-band payment capture, where card data bypasses the AI pipeline and flows directly to a PCI-certified processor.
Pipeline vs. Realtime: Understanding the Core Architectural Choice
When building a production voice agent, you face a fundamental architectural choice: a cascaded pipeline (STT → LLM → TTS) or a realtime speech-to-speech model that handles everything in a single model call.
| Pipeline (STT → LLM → TTS) | Realtime (Speech-to-Speech) | |
|---|---|---|
| Latency | 400–800 ms (streaming) | 200–400 ms |
| Turn detection | Full control — VAD, endpointing, model-based | Built-in only; limited customization |
| Modularity | Fully modular and debuggable | Opaque — audio in, audio out |
| Tool calling | Mature, reliable text-based function calling | Varies by provider |
| Cost | Optimize each layer independently | Difficult to optimize |
| Telephony (8 kHz) | Well-supported with telephony-optimized STT | Trained on 16–48 kHz web audio |
| Compliance | Full data flow control, PII redaction possible | Centralized infrastructure, variable residency |
For contact center, healthcare, and financial services deployments, the pipeline architecture remains the more practical choice — because of compliance, debuggability, and telephony requirements. For a detailed comparison of both approaches, see our article on pipeline vs. realtime architecture.
How to Evaluate a Voice Agent Platform Before You Pilot
The three specs that matter most
1. Production WER under realistic conditions — ask for accuracy data measured with background noise and accented speech. If a vendor won't provide production-condition figures, treat that as a signal.
2. Total pipeline latency end to end — from the caller's last word to the agent's first audio byte. Ask for P50, P90, and P95 percentile data, not just averages. Tail latency is where conversations break.
3. Deployment flexibility — cloud, on-premises, or VPC options that match your compliance requirements. This eliminates vendors before you schedule a demo.
Questions to ask any vendor
- Will you share production latency percentiles (P50, P90, P95)?
- How do you handle domain vocabulary at runtime without retraining?
- Will you sign a BAA if we're in healthcare?
- Can your infrastructure stay out of PCI scope for financial services?
- What are your documented limits for concurrent call capacity?
- How does escalation work — do you pass the full conversation transcript to the human agent?
- Can I replay a specific session and see exactly where something went wrong?
Build your requirements checklist first
Before you evaluate vendors, document your call volume, peak concurrency needs, compliance obligations, and the domain vocabulary your callers use daily. These four variables eliminate most vendors before you schedule a demo — and they prevent you from being dazzled by a vendor demo on clean audio that collapses on your real call recordings.
Conclusion
AI voice agents are not a single product. They are a pipeline of four technologies — ASR, NLU, decision engine, TTS — each with accuracy targets, latency budgets, and compliance constraints that vary by industry and deployment context.
The gap between a convincing demo and a system that works on your actual call data is where most pilots fail. Closing that gap requires testing with production-grade audio, demanding realistic benchmarks from vendors, and understanding the architectural trade-offs before you commit.
Versatik supports you through this entire process: technical audit of your use case, platform recommendations matched to your constraints, production deployment, and continuous optimization. Our teams work daily with the latest voice agent architectures — knowing their real-world performance beyond the benchmarks.
30 seconds to book 30 minutes
Wondering whether AI voice agents can genuinely replace your IVR or handle your intake volume? Our team can help you assess the fit in 30 minutes.