A production voice agent should end with real work completed: an enquiry has an owner, a booking exists in the real calendar, a support case reaches the right queue, or a person receives the context needed to take over. If the conversation sounds natural but the result is missing, duplicated, or wrong, the system has failed.
That distinction shapes how we build these agents. We do not begin with a voice, a character, or a provider dashboard. We begin with the business queue, the decisions inside it, the systems that own the truth, and the point where a human must remain in control.
This guide explains the full operating model we use when designing voice agents with ElevenLabs, Vapi, and the surrounding application stack. It covers where they can be deployed, which professional workflows suit them, how we build knowledge and guardrails, how tools reach real systems, and how we prove the agent works before increasing its authority.
Voice is one part of our wider approach to removing marketing and operational busywork with AI automation. The difference here is real-time conversation: every delay, interruption, tool result, and handoff becomes part of the customer experience.
A voice agent is not valuable because it speaks. It is valuable when it turns a repeated conversation into a safe, owned business outcome.
The useful unit is a governed workflow, not a talking widget. Knowledge answers questions, while deterministic policy and server tools control actions. Web, phone, WhatsApp, and applications can share one core but never the same adapter. And we measure owned outcomes and failure rates, not the number of conversations.
What is a production voice agent?
A production voice agent is a real-time interface to a controlled business workflow. It listens, reasons within a defined scope, retrieves approved information, invokes narrowly authorized tools, and records a trustworthy outcome. It also knows when to stop, transfer, or admit that the available knowledge is insufficient.
The spoken exchange is only the visible layer. Underneath it sit identity, consent, session state, knowledge retrieval, policy, tool authorization, CRM or calendar writes, observability, privacy controls, and fallback handling.
Vapi's current Introduction describes its voice pipeline as speech-to-text, a language model, and text-to-speech. That explains how a voice response is produced. It does not by itself answer whether this caller may change this appointment, whether a lead was persisted, or whether a refund falls within policy. Those are application decisions.
ElevenLabs' current Tools documentation separates client tools, webhook tools, MCP tools, and system tools. That separation is useful because each class has a different trust boundary. Navigating a visitor to a page is not the same as writing to a CRM. Transferring a call is not the same as approving a commercial exception.
Our working definition is simple: the model proposes language and actions; the governed system decides what is true and what is allowed.
Where can voice agents be deployed?
Voice agents can sit wherever a customer or team member already needs to ask, decide, or act. Provider support varies, so we map the channel before selecting the runtime.
ElevenLabs lists React, iOS, Android, React Native, SIP, Twilio, WebSocket, and WhatsApp paths in its current ElevenAgents overview. Vapi documents web calls, phone calls, chat, SMS chat, telephony integrations, and server-side call management across its platform introduction.
We do not describe every provider as natively omnichannel. ElevenLabs currently documents a direct WhatsApp connection for messages and calls. Vapi documents web, phone, chat, and SMS surfaces, but that is not evidence of the same native WhatsApp path. When a desired channel is not native, it needs a separate adapter or a different provider choice.
Which professional workflows are worth automating?
The strongest voice-agent use cases contain repeated intent, approved knowledge, bounded actions, and a clear handoff. They remove coordination without hiding important decisions from the business.
The business outcome matters more than the department label. An "AI receptionist" that only answers FAQs may have less value than a narrow missed-call workflow that creates the right CRM task and confirms ownership. A broad "sales agent" may be riskier than a qualification agent that asks five approved questions and books a person.
How do we choose the first workflow?
We choose the smallest workflow that is frequent, measurable, reversible, and governed by written rules. This is where business thinking prevents technical enthusiasm from creating an expensive demonstration.
We score six conditions:
Then we write an acceptance statement before choosing a provider. For example: "A caller can ask in English or Arabic, confirm a suitable location, book only an available consultation slot, receive an accurate confirmation, and transfer to reception whenever identity, policy, or tool execution is uncertain."
That statement forces useful questions. Which Arabic? Which locations? What proves availability? Can the calendar create duplicates? What happens after hours? Who receives a failed booking? Those answers determine the architecture.
What architecture sits behind the conversation?
We use one governed core with adapters around it. This avoids rebuilding commercial rules for every channel while respecting the technical differences between a browser session, a phone call, and a message thread.
The core owns qualification rules, approved claims, required CRM fields, booking constraints, escalation conditions, and outcome definitions. The adapters own channel identity, microphone state, media types, telephony events, message windows, and transfer behaviour.
This is also why we keep the application server between the provider and consequential systems. Vapi's Server URLs documentation covers status updates, transcript updates, function calls, assistant requests, end-of-call reports, and hang notifications. ElevenLabs provides webhook tools for real-time actions and signed HMAC webhooks for platform events. Those events still need validation, authorization, deduplication, observability, and controlled writes in our application.
The provider runs the conversation. The business application owns the transaction.
How do we build a knowledge base that stays trustworthy?
A useful knowledge base is a maintained publication with retrieval tests, not a folder of PDFs uploaded once. We build it around ownership, scope, freshness, and the decisions the agent is allowed to support.
1. Inventory the source of truth
We list the systems and documents that own services, product facts, policies, locations, pricing authority, eligibility, troubleshooting, and escalation. Each source gets an owner, locale, effective date, and review date. Drafts, internal comments, superseded rate cards, and unapproved sales claims stay out.
2. Separate policy from retrievable context
Hard rules do not belong only in semantic retrieval. Consent requirements, forbidden advice, discount authority, transfer triggers, booking limits, and required confirmations sit in deterministic policy and tool code. The knowledge base explains approved facts. It does not decide whether an action is authorized.
3. Structure content for retrieval
We split large documents into focused topics with clear headings, stable terminology, and enough context to stand alone. English and UAE Arabic are maintained as reviewed source packs. A runtime translation is not assumed to preserve policy, product names, or commercial nuance.
ElevenLabs supports files, URLs, and text in its Knowledge Base. Its documentation also warns that adding one URL does not crawl every linked page or keep the source continuously synchronized. RAG can retrieve from larger indexed sources, but its RAG guide notes an approximate 250 ms latency addition.
Vapi's current Knowledge Base guide uses files and a query tool. Its Query Tool guide requires explicit prompt instructions about when to query, not merely attaching a source. Vapi also supports a custom knowledge base when a business needs its own search, vector store, filters, or privacy controls.
4. Define the unknown state
We write what happens when no source is found, sources conflict, a record is stale, or the question requires judgement. The correct answer may be a clarifying question, a transparent "I need to verify that," or a human handoff. Confidence in tone must never replace confidence in evidence.
5. Test retrieval, not only final wording
The test set includes direct questions, paraphrases, abbreviations, code-switching, misleading premises, similar products, old policy names, and questions that should not be answered. We inspect which source was retrieved, not only whether the final sentence sounds plausible.
6. Connect publishing to review
Knowledge changes need the same discipline as code changes: version, review, publish, evaluate, and roll back. Conversation analysis then identifies repeated unknowns and weak retrieval, which become a content backlog rather than prompt patches.
How do prompts, workflows, and specialist agents fit together?
The prompt defines behaviour, but a production workflow should not depend on one giant prompt remembering every branch. We separate language guidance, deterministic transitions, tool execution, and specialist responsibility.
Our base prompt specifies:
- identity and clear AI disclosure;
- the exact job and topics in scope;
- approved tone, pace, and language behaviour;
- which knowledge must be queried before answering;
- the minimum information needed for each outcome;
- confirmation rules for names, phone numbers, dates, addresses, and actions;
- when to use each tool and how to interpret its result;
- what the agent must never claim or do;
- unknown, failure, and human-transfer states;
- the condition for ending the conversation.
ElevenLabs Agent Workflows provide a graph with subagents, conditions, tool nodes, transfer nodes, and end nodes. A dedicated tool node can guarantee an execution point and branch on success or failure. The workflow JSON can also be managed through its CLI or SDK, which makes review and versioning possible.
Vapi distinguishes Assistants and Squads. A focused assistant is enough for many reception, qualification, and booking flows. Squads split complex work into specialists with context-preserving handoffs. Vapi's Squads guide explicitly frames the trade-off: oversized prompts increase distraction, cost, and latency, while specialists can keep goals and tools narrower.
We use specialists only when the responsibility truly changes. Creating an agent for every branch adds handoff latency and operational complexity. One focused assistant with clear tools is usually the better starting point.
How do tools become safe business actions?
Every consequential action passes through a narrow server contract. The model supplies proposed arguments; the server authenticates the request, authorizes the action, validates the payload, applies business rules, writes once, and returns a typed result the agent can truthfully describe.
The endpoint returns states such as created, updated, duplicate, requires_confirmation, not_authorized, temporarily_unavailable, or handoff_required. The agent does not translate a timeout into success. If a CRM accepted the write but the response was lost, the idempotency key prevents a retry from creating a second lead.
We separate tools by consequence:
ElevenLabs Webhook Tools support stored secrets, bearer credentials, basic authentication, custom headers, and OAuth2 options. Its client documentation correctly directs server-side API work to webhook tools. Vapi's Server Authentication supports reusable bearer, OAuth2, and HMAC credentials. Vapi also documents tool argument encryption for selected sensitive fields.
Secrets never belong in a browser bundle or a prompt. A public web key starts a session; a private server key performs privileged provider operations.
That server boundary is one reason developers directly affect marketing performance: attribution, consent, reliability, and data quality are commercial concerns expressed through implementation.
What guardrails do we implement beyond the prompt?
Guardrails work as layers because no single layer understands every risk. Provider controls can keep a conversation in scope. Only the application knows whether this caller is authorized, whether this slot is still available, or whether a commercial exception is permitted.
Our baseline covers:
- Disclosure and consent: say that the user is interacting with an AI system and separate service consent from recording or follow-up consent.
- Scope: define allowed topics, forbidden advice, and the exact actions available.
- Input protection: detect prompt injection, abusive patterns, and unnecessary sensitive information.
- Knowledge grounding: require approved retrieval for business facts and define the unknown state.
- Tool allowlists: expose only the tools required for this workflow with strict schemas.
- Server authority: authenticate, authorize, validate, constrain, and log every consequential action.
- Confirmation: repeat critical fields and confirm irreversible or high-impact changes.
- Human escalation: transfer uncertainty, vulnerability, anger, regulation, exceptions, and explicit requests.
- Data controls: minimize collection, redact where appropriate, restrict access, and set retention deliberately.
- Operational controls: rate limits, anomaly alerts, version rollback, provider fallback, and a tested kill switch.
ElevenLabs' current Guardrails guide separates prompt hardening, user-input validation, and response validation. It also notes that its custom guardrails are in alpha and that blocking validation adds latency. We therefore test guardrail behaviour as a dependency, not as an unquestioned guarantee.
For web access, ElevenLabs' Agent Authentication recommends signed URLs for authenticated client sessions. Those URLs expire after 15 minutes. Domain allowlists are a separate option, and the documentation says not to configure both on the same agent.
For event receivers, we verify signatures or configured credentials before parsing business data. We also protect against replay, duplicate delivery, and an event arriving after the user has already retried through another channel.
How does deployment differ by channel?
The core workflow can be shared, but production behaviour changes materially across the web, phone, WhatsApp, and embedded applications.
Website and product agent
On a Next.js site, we keep the page useful without voice. The browser integration loads as an optional client boundary after intentional interaction or a controlled delay. Navigation remains a client tool. Lead capture, booking, account lookup, and communications go through authenticated server routes.
The agent receives only the page, locale, campaign, and user context required for the task. Microphone permission is requested at the moment of use. The UI shows connection, listening, speaking, tool, transfer, and failure states rather than hiding them behind one animated orb.
Our current TNM website integration proves deferred loading, locale-aware navigation, and a client-tool contract. Its lead tool still records a Phase 0 payload locally. We do not describe that as CRM persistence. Production acceptance requires a verified server write, idempotency, durable failure handling, and an accountable record owner.
ElevenLabs provides a React SDK. Vapi's Web Calls guide separates the public web key used by the client from the private token used by its server SDK.
Phone agent
Phone deployment adds number ownership, carrier behaviour, caller identity, call recording, voicemail, interruption, transfer, concurrency, and regional availability. We test the real telephone route, not only a browser preview.
ElevenLabs documents SIP trunking and native Twilio connectivity. Vapi supports imported telephony and SIP options in addition to its phone-number flows. Existing numbers, caller ID, inbound routing, transfer type, and recovery behaviour are decided before launch.
ElevenLabs' current WhatsApp guide covers inbound and outbound messages, audio messages, calls, templates, and call-permission requests. An authorized WhatsApp Business account and correct account ownership remain part of the deployment.
We also decide whether another Meta application already owns templates, routing, consent, and support. Two automated responders on one number create conflicting state. Human takeover must suppress later automation, and opt-out must update the system of record.
Native, embedded, and internal applications
Authenticated application surfaces can carry richer context than a public page, but that does not justify broader authority. User identity, account role, current object, and source permissions are passed deliberately. Client tools may update local UI state, while business changes still route through the same server contracts.
How do we design human handoff?
Handoff is an expected state with its own data contract. The agent should transfer before it becomes trapped, not after several apologies and repeated questions.
We define triggers for:
- the user asking for a person;
- low confidence or conflicting knowledge;
- tool failure or repeated validation failure;
- regulated, medical, legal, financial, or safety-sensitive topics;
- a vulnerable, distressed, or angry customer;
- commercial exceptions or authority limits;
- identity uncertainty;
- any action outside the workflow's written scope.
The receiving person gets the available identity, reason for contact, concise summary, collected fields, sources used, tools attempted, results, failure state, and requested next action. The user is told what is happening and what to expect.
ElevenLabs' Transfer to Number supports conference, blind, and SIP REFER transfer patterns, with feature differences between Twilio and SIP. Its warm-transfer message is not available in every transfer type. Vapi's Handoff Tool can route to another assistant, squad, or dynamic destination and control how much prior context is forwarded.
The transfer design follows the destination system. A phone transfer, a live-chat takeover, and a CRM callback task are different operational promises.
How do we test the agent and measure business value?
Testing has four layers: deterministic contracts, conversation behaviour, end-to-end audio, and production outcomes. A prompt preview covers only part of one layer.
ElevenLabs' Agent Testing supports scenario, tool-call, and multi-turn simulation tests. Vapi's current Evals validate mock conversations and tool calls, while Simulations run realistic conversations. Vapi's older Test Suites are documented as being deprecated, so we do not build a new testing process around the old interface.
After a conversation, we need structured outcomes, not only transcripts. ElevenLabs Conversation Analysis supports success evaluation and data collection. Vapi Structured Outputs can extract validated JSON from the full call context, and its scorecards can grade numeric or boolean outputs.
Business reporting pairs an outcome with a protection metric:
Conversation volume is operational load. It is not business value by itself.
How do we choose between ElevenLabs and Vapi?
We choose the provider after defining the workflow, not before. Both platforms can support serious voice systems, but they expose different primitives and product surfaces.
We then test interruption, accents, Arabic-English switching, tool latency, provider failure, transfer, event delivery, and the required data policy. The cheapest or most feature-rich provider on paper can be the wrong choice if it does not pass the actual channel and operating constraints.
Provider portability also matters. Prompts, knowledge sources, tool contracts, evaluation cases, and business outcome definitions should live in versioned project assets where practical. The provider configuration is important, but it should not become the only copy of the company's operating logic.
What privacy and outreach controls are required?
Voice systems handle identity, speech, transcripts, recordings, and business records, so privacy is an architectural decision. We decide what is collected, why it is needed, where it is stored, who can access it, how long it remains, and how a user can reach a non-recorded or human path where required.
ElevenLabs documents configurable retention and per-agent Zero Retention Mode. ZRM changes the debugging model because limited call data remains available, so required post-call webhooks and internal records must be designed deliberately.
Vapi's current recording consent plan can place a consent assistant before the main assistant for eligible enterprise configurations. Vapi's newer artifactPlan controls recording, logging, and transcripts; its documentation notes that older recording properties are deprecated.
For outbound workflows, we do not interpret a provider's batch-call or campaign feature as legal permission. We start with expected, documented contact such as a requested callback, appointment reminder, service update, event follow-up, or eligible renewal. Consent or another valid basis, suppression, opt-out, identity, contact timing, recording, platform rules, and current UAE legal review are launch requirements, not copy added later.
This article describes our engineering approach, not legal advice. The deployment must be reviewed against the business, jurisdiction, channel, and current rules in force.
What does production acceptance mean to us?
Production acceptance means the workflow can complete valid work, fail visibly, transfer safely, and produce evidence an owner can review.
Before increasing traffic or authority, we want:
- a written business job, owner, outcome, and protection metric;
- approved English and UAE-Arabic sources with ownership and review dates;
- explicit disclosure, consent, recording, retention, and outreach rules;
- a versioned prompt and workflow with a defined unknown state;
- browser-only actions separated from authenticated server tools;
- schema validation, authorization, idempotency, action limits, and audit logs;
- a visible retry or manual queue for accepted work that cannot reach its system;
- tested transfer, provider-failure, and non-AI fallback paths;
- restricted transcript, recording, and tool-trace access;
- regression cases for normal, ambiguous, hostile, multilingual, and failed-tool scenarios;
- monitoring, alerts, version history, rollback, and a tested kill switch;
- reporting tied to owned business outcomes rather than conversation volume.
The smallest useful deployment is usually one workflow, one clear outcome, and one accountable team. Once it performs reliably, the same governed core can expand to another channel or job without turning into an all-purpose agent nobody can reason about.
If you have a queue of missed enquiries, booking work, support intake, requested callbacks, or repeated customer questions, our AI and automation work starts by mapping the process and its failure states. We then decide whether voice is the right interface and which provider can pass the acceptance test. Talk to our team.











