The voice transforms waiting into silence, error into interruption, and ambiguity into potentially irreversible action. A voice agent must understand imperfect audio, manage turns of speech, consult tools, and respond with sufficiently low latency to remain natural. It must also allow the user to correct, to request a human, and to know that they are speaking to an AI.
Speech-to-speech models and real-time APIs have progressed in 2026. Production remains a system problem: media, network, identity, policies, tools, supervision, and contact center.
Models, prices, regions, and voice functions must be confirmed in the official documentation before implementation.
Define the role of the agent
The uses can be:
- reception and guidance;
- qualification;
- appointment scheduling;
- consultation of a statute;
- internal assistance;
- structured collection;
- reminder;
- processing of simple requests.
High-impact actions — payment, contract modification, regulated advice — require authentication, confirmation, and sometimes a human.
The script must specify what the agent does not do and how he transfers.
Two audio architectures
Cascading pipeline
Voice recognition, text, LLM, synthesis. Each component is controllable and replaceable. The chain adds latency and may lose prosodic information.
Speech-to-speech model
The model processes and generates audio in a more integrated way, sometimes full-duplex. The conversation can be more natural. Control over transcriptions, voice, and structured outputs must be provided.
A hybrid architecture can maintain a working transcription and tools structured around a real-time model.
Transport and session
WebRTC is suitable for real-time browsers and applications; telephony can go through SIP or a provider. The session must manage:
- media negotiation;
- codecs;
- packet loss;
- jitter ;
- regional routing;
- resumption;
- authentication;
- encryption;
- end of appeal.
Client-side tokens are short and limited. Platform secrets remain on the server side.
Build a latency budget
The experience depends on the delay before a sign of listening and the delay before a useful response. The steps are:
- incoming transport;
- speech detection or continuous understanding;
- inference;
- tool;
- audio generation;
- outbound transport.
Pipeline of a voice agent from audio transport to tools, to response and human transfer.
Manage speaking turns
A classic system waits for silence to decide that the user has finished. Silence that is too short cuts off; too long slows down. Full-duplex models can listen and speak simultaneously, but must handle interruption and overlap.
The user must be able to interrupt. The current audio stops, the state is updated, and unconfirmed actions are canceled. The system distinguishes a noise from a correction.
Visual or auditory cues indicate listening, processing, and transfer without simulating a deceptive human presence.
Do not hide tool times
A CRM call or a search can take several seconds. The agent can confirm the task, provide accurate intermediate information, or offer a callback. They must not invent a filler conversation.
The tools have timeouts, retries, idempotence, and state. A long operation becomes asynchronous, and the result is confirmed when it is real.
Authenticate without degrading the experience
The voice alone is not sufficient proof of identity. Depending on the risk, use:
- known non-sensitive information;
- one-time use code;
- link in an authenticated channel;
- application connection;
- transfer to an advisor;
- reauthentication before action.
The agent never asks for a completely unnecessary secret. The answers are hidden in the logs and the speech synthesis.
Confirmation of actions
Before an action, the agent repeats the essential parameters: object, date, amount, recipient or consequence. The confirmation concerns this version and expires.
For data that is difficult to understand orally, a written summary can be sent in a secure channel. The user can correct a single field without restarting the entire conversation.
Consent and transparency
The call clearly indicates the use of AI and the relevant purposes. If the conversation is recorded or transcribed, the information and the applicable basis must be established with the competent authorities.
Data collection is minimized. Transcription, audio, summary, metadata, and feedback may have different durations. The choice of voice must not imitate a person without authorization.
Transcription and sensitive data
The transcription may contain name, health, payment, or secret. Access, retention, and masking are defined. The data is not sent to unauthorized tools.
A transcription confidence score may trigger a request for repetition. Proper names, addresses, and references are confirmed, sometimes by spelling or text channel.
Design the tools
Voice tools must be short and structured. The model does not read a raw technical response. A layer transforms the statuses into exact and safe sentences.
The actions are limited, audited, and verified. The model cannot freely choose an external recipient or expand its rights.
Human transfer
The transfer must be accessible at any time or according to rules. The advisor receives, with consent and appropriate rights:
- reason;
- steps completed;
- identity verified;
- collected data;
- uncertain points;
- pending actions.
The user does not repeat their entire story. If no advisor is available, the system offers a real alternative.
Degraded mode
In case of noise, weak signal, unavailable model, or slow tool:
- to ask to repeat;
- switch to keyboard/text;
- limit actions;
- transfer;
- create a reminder;
- finish cleanly.
A failure must not leave an action in an unknown state.
Evaluate the quality
The test set includes:
- accents;
- noise;
- overlap;
- interruptions;
- numbers;
- names;
- ambiguities;
- emotion;
- languages;
- refusal;
- tools;
- transfer.
Measure task success, corrections, time, p95 latency, transfer rate, critical errors, satisfaction, and cost. Evaluators listen to authorized excerpts according to a policy.
Real-time observability
Monitor network quality, packets, latency, towers, interruptions, tool calls, errors, disconnections, costs, and escalations. The session ID links media and business without storing more than necessary.
Alerts detect an increase in silence, abandonment, or repetitions.
Cost
The cost combines incoming/outgoing audio, model, telephony, tools, storage, transcription, supervision, and human transfer. It is related to a completed task.
Routing can reserve the most capable agents for complex conversations. Non-urgent calls can use an asynchronous callback.
A gradual deployment
Start with a simple pattern, with limited schedules and population, then analyze each failure. Sensitive actions remain as proposals. Capacity expands when latency, quality, and transfer are controlled.
Partitech can design the real-time platform, integrate business tools, secure identity, and build the evaluation campaign. The goal is a useful and transparent voice, capable of acting without sacrificing control or quality of service.
Let's talk about your project
Prototyping and industrializing a business voice agent with Partitech. Contact Partitech.