Voice AI demonstrations are often convincing. A conversation is transcribed within seconds. Speakers are separated. A summary appears. Weak signals are extracted. The effect is immediate, and so is the urge to deploy.
But between a successful demo on a clean recording and a system that is truly production-ready at the scale of a customer contact centre, the gap is significant. The challenge is not just transcription. It is turning an imperfect, noisy and sometimes ambiguous voice stream into structured, reliable information that can be integrated into the information system.
Speech-to-text is only the first layer
Accurate transcription is necessary. It is not sufficient.
Current engines — Whisper, Azure Speech, Deepgram or AssemblyAI — deliver impressive results on studio recordings or well-captured conferences. But in a call centre, conditions are rarely ideal. Recordings go through 8 kHz mono telephony with compression, background noise, regional accents, overlapping speech, network dropouts and domain-specific terms absent from the model’s standard vocabulary.
A system displaying a word error rate (WER) of 5% on a clean English benchmark can easily climb to 15 or 20% on real-world French calls with accents and sector-specific jargon. And on certain critical segments, a single misspelled proper name or contract number is enough to render the rest of the chain unusable.
The first task is therefore to measure transcription quality under your actual conditions, not on a demo dataset provided by the vendor.
Diarisation changes everything
For many use cases, knowing “who said what” is just as important as the text itself.
Take a concrete example: a Voice of the Customer analysis in insurance. The objective is to identify the sources of dissatisfaction expressed by policyholders during calls to the claims department. Without robust diarisation, the system can attribute to the customer a sentence spoken by the advisor, or vice versa. A customer’s objection becomes an advisor’s rephrasing. A commitment made by the operator vanishes into the flow.
Diarisation is a notoriously difficult problem, especially on single-channel recordings. Specialised models such as pyannote.audio have made significant progress, but the diarisation error rate (DER) remains sensitive to audio quality, turn duration and moments of overlap. On short, fast-paced conversations — like a complaint call — these moments are frequent.
This is not a technical detail. It is a quality prerequisite for all downstream uses: reason classification, dissatisfaction detection, advisor quality assessment.
The real challenge is producing exploitable structure
The value of a Voice AI system does not come from raw transcription. It comes from the structure you derive from it.
In a Voice of the Customer programme in telecoms, the goal is not to have the full transcript of 10,000 calls per month. The goal is to extract recurring contact reasons, persistent irritants, objections that advisors struggle to address, unfulfilled commitments, and early signals of churn.
This requires structuring work that goes well beyond speech-to-text: business-adapted reason taxonomies, clear qualification rules, well-defined use cases and evaluation datasets that measure classification stability over time. A system that correctly detects the reason “billing issue” every other week is not production-ready, even if its transcription is perfect.
In other words, the system must turn sound into actionable elements, not just text.
Integration matters as much as the model
A Voice AI system is only valuable if it fits into the rest of the information system. If the output remains a summary in a shared file or a CSV export sent manually every week, the gains will quickly plateau.
In practice, integration means connecting the system:
- to the CRM, to link each call to a customer record;
- to the quality tool, to feed evaluation grids;
- to the ticket database, to correlate voice-based reasons with written complaints;
- to dashboards, to surface trends for managers.
Each connector adds considerations around access rights, logging and error handling. If the CRM is unreachable for two hours, what does the system do? If a call contains health data, how is it isolated before storage? The architecture must provide for the connectors, but also for retry mechanisms, retention rules and separation between test and production environments.
The challenge is not only technical, it is also evidentiary
As soon as voice data starts feeding a decision, a quality assessment or a customer action, the exact role of the system in the decision chain must be clarified.
A system that helps a supervisor identify which calls to review first does not have the same status as a system that automatically scores an advisor’s quality. The former is an assistance tool. The latter is a decision-making tool, with implications for employment law and social dialogue.
Many problems start when an assistance tool is tacitly treated as a decision-making tool. The “indicative” quality score becomes the basis for the annual review. The “suggested” dissatisfaction detection becomes the criterion for a customer callback. Without explicit framing of what the system can and cannot decide on its own, these shifts are almost inevitable.
What to measure
Evaluating a Voice AI system requires more than an overall transcription rate.
You should look in particular at:
- transcription quality under your own audio conditions, not on a standard benchmark;
- diarisation quality for your actual call durations and conversation profiles;
- classification stability over time, because a model can drift as contact reasons evolve;
- the rate of ambiguous cases, to properly size human review;
- the errors that cost the most, identified and tracked specifically.
The right system is not the one that impresses on a hand-picked extract. It is the one that remains readable, stable and useful when volume scales to several thousand calls per day.
Conclusion
Production-ready Voice AI starts where simple transcription ends. The real challenge is the quality of the structure produced, the robustness of diarisation, business integration and clarity about the role assigned to the system in the decision chain.
General framework: Controlled AI, our doctrine for engineering AI systems in real-world environments.
Distinguishing what belongs to the demo from what can actually be industrialised: software development and managed services for trust-critical systems.