2025 produced a lot of AI activity. It also exposed where CX breaks.

As Voice AI moved from pilots to real deployments, the gaps became hard to ignore. Some approaches scaled, others added friction, and many revealed that the hardest problems in CX still live inside live conversations.

This roundup surfaces the shifts, data, and ideas that matter heading into 2026.

1. The State of Voice AI

Voice didn’t just improve in 2025. It showed the industry where CX actually breaks. This report highlights where legacy systems, weak adoption, and language gaps continue to create cost and inconsistency across contact centers.

Read this to understand how voice AI is being used today and where teams still struggle to scale it effectively.

2. 2026 Voice AI Productivity Predictions

2026 won’t be about more AI. It will be about where AI holds up in live conversations and at scale. These predictions focus on where Voice AI reduces friction in real time, improves understanding, and helps agents resolve issues faster without breaking trust.

Read this to understand what actually drives Voice AI productivity and why clarity, not automation, is the lever that scales.

3. From Fragmentation to Focus: A Playbook for Eliminating Agent Burnout

Burnout is a downstream symptom. The real problem is fragmented CX systems, misaligned metrics, and automation pushed too far.

This guide breaks down how today’s contact center stacks create cognitive overload and where purposeful, real-time Voice AI can support agents without removing human judgment from the conversation.

Read this to understand why burnout shows up operationally and what changes actually reduce agent load.

Get the Playbook

In the News

4. Why 95% of AI Pilots Fail and What the 5% Do Differently

This isn’t about models falling short. It’s about how AI is deployed. The companies that succeed aren’t leading with autonomous, customer-facing agents. They’re starting with co-pilots that augment humans, where trust is built into the workflow. This piece lays out what separates experimentation from production.

Read this to understand why copilots scale, autonomous agents stall, and why deployment choices matter more than ambition.

Read the full story

5. The State of CX: What 2025 Taught Us

Customers now expect more than efficiency. In 2025, AI moved from experiment to expectation, but the gap between brand promise and delivery widened. Teams advanced automation, personalization, and data use, but cracks showed up in empathy, voice of the customer, and emotional connection.

Read this to understand where CX investments outpaced real customer experience and why connection still matters as much as capability.

Read the full story

6. 48% of CX Leaders Plan to Access AI via BPO Partners

As Voice AI adoption accelerates, BPOs are playing a larger role in making AI usable at scale and are often better positioned to operationalize it inside live contact center environments. Rather than building everything in-house, teams are turning to BPOs to reduce risk, speed deployment, and embed AI into real workflows.

Read this to understand why Voice AI adoption is shifting toward partners who can operationalize it, not just vendors who sell it.

Read the full story

Worth Watching

7. Accent Conversion’s 85+ NPS Impact

A concrete example of what happens when you remove friction from voice conversations at scale. The outcome wasn’t marginal. It was structural.

Watch this to see how improving comprehension in real time can drive measurable CX outcomes, not marginal gains.

8. Inside the Data: The State of Voice in CX Unpacked

The session also digs into where deployments fail, why overpromising slows adoption, and how measurable outcomes are replacing futuristic demos as the bar for investment.

Watch this to understand where voice AI is delivering real value today and why pragmatism, not hype, is driving the next phase of CX adoption.

9. The Rise of Voice Productivity

A conversation on why voice remains the highest-stakes channel in CX and how teams are redefining productivity beyond speed alone. Voice productivity is about reducing friction in live conversations. When clarity improves, teams see fewer repeats, faster resolution, and better outcomes for both customers and agents.

Watch this to understand why clarity, not automation, is the real driver of voice productivity.

Watch now

10. What We’re Carrying Into 2026

• Voice quality drives outcomes. When conversations break down, everything slows: resolution, satisfaction, and agent capacity. Teams that invest in clear, reliable conversations resolve issues faster, protect CSAT, and give agents more capacity to do real work.

• Language and accent friction is expensive. The cost shows up in repetition, longer calls, and churn long before it shows up in reports. When understanding improves, calls shorten, repeats drop, and loyalty increases across global customer bases.

• AI creates value in the moment. The biggest gains come from supporting agents during live interactions, not from adding more layers of automation. When AI reduces friction in real time, agents stay focused, errors drop, and customers get to resolution faster.

If 2026 is about anything, it’s this: CX improves when conversations get easier for the people on both sides of the call.

Voice AI in 2026 is about real-time clarity, not more tools.

Over the past year, we’ve seen a sharp rise in real-time voice adoption across contact centers, collaboration tools, and consumer AI. More than 80 percent of contact center leaders now list AI productivity as a top priority. Models are faster, latency is dropping, and the gap between human and machine interactions is shrinking.

At Krisp, we process over 80B minutes of voice every month and work closely with enterprises, BPOs, and developers building the next generation of voice-driven applications. These predictions come from what we see in the market, the problems customers are trying to solve, and the technology that’s now mature enough to deploy at scale.

Here’s where Voice AI Productivity is headed in 2026.

1. Call center agent copilots become the new standard

AI copilots move from early adoption to everyday use.

In 2025, real-time call guidance grew faster than any other contact center AI tool. In 2026, copilots will be part of every agent’s workflow, giving them live prompts, better clarity, and instant context.

This shift is driven by three forces:

• Rising cost pressure on service teams

• Higher expectations from customers

• Lower cost and higher quality of real-time STT

By the end of 2026, many global call centers will deploy copilots across entire programs, not just pilot groups.

The impact is fewer escalations, shorter handle times, and more consistent service from every agent.

2. Language barriers disappear at work

We enter the first real year of “global teams sounding local.”

Real-time accent conversion and language translation get embedded directly into meeting tools, contact center platforms, and collaboration apps. Latency drops. Accuracy improves. And companies start expecting voice clarity as a built-in feature, not an add-on.

Today, language issues contribute to over 20 percent of repeated calls in offshore teams. With better translation and accent clarity, clarity becomes the new driver of global productivity.

The impact is simple:

• Fewer misunderstandings

• Faster resolutions

• Improved and deepened customer trust

• A more inclusive workplace

By 2026, most major communication tools will ship with real-time translation or accent clarity as standard capabilities.

3. Agent coaching moves from training rooms to live calls

The model for agent development flips.

Traditional coaching cycles rely on manual scorecards, delayed feedback, and hours of formal training. In 2026, the first layer of coaching will happen inside the call itself.

Real-time AI will:

• Provide next-best-action prompts

• Catch compliance violations

• Offer guidance on tone or empathy

Early data from call centers using real-time coaching shows 10 to 20 percent improvements in AHT and CSAT within weeks (based on aggregated benchmarks from Krisp deployments and public case studies from Cresta, Cogito, and Balto).

Companies will shift training budgets from retraining to reinforcement because AI now handles the micro-corrections that drive performance.

4. Call Centers turn voice data into a new revenue line

2026 is the turning point for outsourced contact centers.

Historically, Call Center BPOs haven’t had full access to transcripts or voice data due to security restrictions. With privacy-safe architectures like Krisp’s, BPOs can finally apply AI across every seat without moving or storing sensitive audio.

This unlocks a new business model:

• Real-time speech analytics

• Predictive insights for clients

• Program-level performance dashboards

• New premium-tier service offerings

BPOs shift from cost-driven execution to insight-driven value. By the end of 2026, the top global BPOs have speech analytics and AI-enhanced insights as core differentiators.

5. Meeting productivity shifts from note-taking to real-time intelligence

AI meeting assistants grow up.

Millions already use AI note-takers, but transcription alone is no longer enough. Teams want accurate insights while the meeting is happening, not after the fact.

In 2026, AI becomes the second brain in every meeting.

It will:

• Detect decisions as they’re made

• Track action items across teams

• Summarize long discussions into structured, usable outputs

• Improve clarity for global teams with localization, insights, and context

The shift is already underway. Platforms already process millions of summaries per month. This pace only accelerates as real-time intelligence becomes the baseline expectation.

The bottom line

2026 is the year Voice AI becomes infrastructure.

The winners will be the teams that focus on measurable outcomes, not experiments. Real-time clarity, copilots, translation, and meeting intelligence are no longer emerging areas. They are becoming required capabilities for modern work.

Companies that invest now will transform their service quality, workforce productivity, and global collaboration. Companies that wait will fall behind and fast.

When people talk about AI and automation, they often forget where customer experience truly happens — in conversation.

Voice is still how trust is built, problems are solved, and loyalty is earned.

That’s why the launch of Accent Conversion for Africa is about more than product.

It’s a signal of how Voice AI is reshaping global communication, from the sound of support to the scale of inclusion.

A rising CX powerhouse

Africa is quietly becoming one of the most important regions for CX support.

• Global enterprises are setting up operations across South Africa, Kenya, Uganda, and Nigeria.

• The workforce is young, skilled, and speaks English.

• Cultural alignment with Western markets is strong.

But even in the best contact centers, accent gaps still cause friction: longer calls and missed opportunities.

Technology that changes the conversation

Krisp Accent Conversion for Africa bridges those gaps in real time:

• Converts African English accents to a neutral American English

• Keeps each speaker’s authentic voice and tone intact

• Runs securely on-device with no lag or data exposure

This means agents don’t have to change who they are to be understood.

Clarity becomes universal, not cultural.

Why this matters globally

Accent Conversion is part of a broader shift in Voice AI infrastructure, one that makes the voice channel more inclusive and scalable.

It removes the need for accent-neutralization training, saving time, cutting costs, and expanding access to high-value CX jobs across Africa. It also eliminates the cognitive load offshore agents carry to do their day-to-day jobs.

It’s the foundation of a more connected and fair voice economy where:

• Talent isn’t limited by geography or accent

• Companies can hire globally without communication barriers

• Customers experience clarity, empathy, and speed in every call

When every call is understood the first time, efficiency and satisfaction rise together.

A broader shift in global CX

Africa’s CX evolution mirrors what we’ve seen in India, the Philippines, Pakistan, and Latin America: a combination of human skill, linguistic diversity, and now, expanded reach with voice AI.

With Accent Conversion, Noise Cancellation, and Voice Translation working together, Krisp is building the core Voice AI layer for the modern contact center to help African BPOs compete with the world’s leading hubs.

Even in an AI-driven industry, it’s people who build trust, and AI that ensures they’re heard clearly.

What’s next

Voice AI is entering an era where clarity, empathy, and speed define great customer experiences.

Accent Conversion for Africa marks another step toward frictionless voice.

Read the Full Announcement

On Oct 2nd, I attended the very first VapiCon organized by Vapi.

This was by far the largest Voice AI gathering so far. Not my words, but pretty much everyone was asserting this.

I heard the demand turned out to be much bigger than they could fit.

The keynote speakers did a great job laying out the state of Voice AI.

Big kudos to Jordan@Vapi, Scott@Deepgram, Justin@OpenAI and Dylan@Assembly.

Also big kudos to the event staff. They were very caring and accommodating.

Unfortunately, I had to miss a lot of sessions but here are my takeaways from the ones I did attend:

• The Voice AI community is vibrant and energized. It’s rare to see so many young folks focused on an industry. The future is clearly being built here.

• I think most of the attendees were developers from startups. I estimate that there are between 500-1000 startups in Voice AI now.

• $2B+ has been invested in startups since 2024.

• STT accuracy, turn-taking, speaker-separation, latency, TTS accuracy, hallucinations, function-calling, lack-of-context are the main technical challenges the industry is trying to solve.

• Most of these issues have significantly improved in the last 2 years but they all STILL remain big challenges (except maybe latency?)

• There is a big debate about speech-to-speech (S2S) models and the Cascading approach (STT→LLM→TTS). There are clear pros and cons here.

• The vast majority of the deployed technology uses the Cascading approach

• It is assumed that most of the technical problems will go away once S2S models mature, but this is yet to be seen.

While the event was super energized, I couldn’t help but think about a topic that was in the air. I think people are NOT talking about this enough, while everyone does know about it.

Most Voice AI Agents in production are still quite unstable today.
And this SLOWS the industry down.

I think there are 2 main reasons why they keep failing:

1. The real world is complex: there is too much context and too many edge cases out there, and AI agents simply don’t know how to handle these yet. In contrast, people are really good at this because of our experience, context and multi-modality.
   During VapiCon, I witnessed several live demos fail because of these exact issues—background noise, hallucinations, and AI agents not taking turns properly. It was a clear reminder that even advanced systems still struggle when faced with messy, unpredictable real-world conditions.

2. STT is failing us: In most cases STTs do a great job of capturing what people said. However, the error rate of properly capturing numbers, emails, addresses, and nouns remains quite high. This might seem like a small issue but it’s not, especially in B2B use cases. Perhaps it’s the most important technical problem today, preventing the industry to grow. I really do hope this will get better in 2026.

I estimate the Voice AI Agents traffic to be around 3B/mins/month today.

If we solve the above two problems, the traffic will skyrocket 🚀

Here is to reaching 100B/mins/month by next VapiCon!

Voice AI in CX: What 819 Leaders Reveal About the Future of Voice

Contact centers are under more pressure than ever. Rising labor costs, demanding customers, and the complexity of serving global audiences are forcing operators to rethink their playbook. Krisp, in partnership with Ryan Strategic Advisory, surveyed 819 enterprise CX leaders and one thing is clear: AI isn’t a “someday” investment—it’s a requirement in 2025.

This research highlights how enterprises are navigating the transition from legacy systems to AI-powered voice tools, and what risks they face if they wait too long.

Most are still stuck in transition

The data shows a sharp divide. Many contact centers have adopted modern AI solutions like noise cancellation software, translation, or accent conversion, but most still rely heavily on labor-intensive fixes like human translators or expensive hardware.

That patchwork approach creates hidden costs:

• Customer experience suffers from slow or inconsistent service

• Agents juggle tools instead of focusing on conversations

• Quality varies across regions and languages

• Costs stay high as human services and hardware pile up

A wave of new adoption is coming

The next 6–12 months will be decisive. Large portions of CX leaders plan to roll out AI accent reduction, translation, and noise suppression in the short term. This signals a market ready to modernize voice infrastructure fast.

But while some are preparing to leap ahead, others are holding back, waiting for “perfect” solutions. That hesitation could prove costly.

BPOs are the shortcut—for now

Nearly half of enterprises say they’ll access AI through BPO partners in the near term. It’s seen as the fastest path forward, but it comes with trade-offs: less control, less flexibility, and higher long-term risk.

Those that invest directly in AI will be better positioned to reduce costs and build sustainable, differentiated customer experiences.

Budgets tell the real story

The survey also reveals where the pain is sharpest:

• Over 50% of all voice budgets still go to onshore live agents

• Offshore staffing, automation, and AI account for much smaller shares

This imbalance shows how much potential savings and efficiency are still on the table if AI can close the accent and language gap.

What this means for 2025

• AI is entering a mainstream deployment phase—waiting is no longer safe.

• Accent and language barriers remain the toughest challenge.

• BPOs are filling the AI gap, but enterprises that own their AI future will win.

• Onshore agents drive the cost burden, making offshore + AI an attractive path forward.

AI in CX is no longer hype. It’s a practical tool that’s already reshaping budgets, strategies, and outcomes. Leaders who act now stand to gain speed, lower costs, and a lasting competitive edge.

Read the full report

Next week on the podcast

Ryan Strategic Advisory takes us inside the data on Voice AI adoption to unpack what’s real and what’s hype.

Krisp’s Accent Conversion technology has been on a rapid path of continued innovation since v3 launched in March 2025, when it became mature enough for wide-scale deployment.

In just a few months, the technology has delivered major advancements:

• May 2025, v3.5: Delivered a 20% quality boost for Filipino English and Indian English accents. Rolled out to 95% of Krisp desktop users in two days with strong agent and customer feedback.

• July 2025, LATAM Pack: Added Latin American English accent support, now deployed across thousands of agents.

• August 2025, v3.7: Focused on the Indian English accent pack, delivering significant gains in naturalness, voice stability, and clarity.

What’s New in v3.7

• 14% naturalness improvement (expert-rated) with more human-like speech and better handling of filler sounds

• Improved voice stability with smoother pitch and tone, especially for thick accents

• 5% clarity boost with fewer artifacts and more intelligible speech

• Better pronunciation accuracy with a 4% reduction in phoneme errors, improving difficult sounds like “R” and “L”

How It Was Measured

• 78 real-world agent call samples evaluated

• Expert panel scoring, crowdsourced listening tests (3,120 votes), and objective metrics from Meta Audiobox Aesthetics and Facebook NN Phonemizer

• Crowdsourced preference showed v3.7 was chosen 20% more often for sounding more natural

Read the full technical deep dive with benchmark data and audio samples.

Read the Technical Article

Why It Matters

Accent Conversion v3.7 raises the bar for voice naturalness in high-variance accents, addressing key call center challenges around intelligibility and customer experience.

At CCW Vegas 2025, it was clear that AI alone isn’t the story; AI that elevates human experience is.

Across sessions and show floor conversations, the most forward-looking leaders weren’t chasing the latest hype. They’re focused on strategic, human-centered approaches to AI in CX.

AI + Humans: Collaboration, Not Competition

AI tools work best when they enhance the strengths of human agents. Leaders across the industry are rethinking AI’s role: from automation-first to collaboration-first.

The goal is no longer to replace agents but to help them deliver better, faster, more empathetic customer experiences—while improving agent experience, too.

Strategy First: Align AI with Human Touchpoints

Successful AI investments start with a clear strategy. Success means aligning AI tools with the right moments in the customer journey, maintaining transparency with customers, and actively supporting employees through adoption.

Without this foundation, even the most advanced AI can fall flat.

The ROI Reality: It’s in the Rollout

How you implement and deploy AI determines whether you see ROI. Sessions highlighted that thoughtful rollout—designed for real gains in CSAT, first-call resolution, and agent productivity—is what separates success from wasted spend.

A poor deployment can erode trust and negate value.

Voice Still Rules

Even in an increasingly digital-first world, voice is the highest-stakes channel for support. Complex, emotional, or high-value interactions still rely on human voice and interactions.

Innovations that remove friction and bridge communication gaps are helping make voice an even stronger channel.

Agentic AI: Still a Gray Area

Agentic AI was a buzzword at CCW, but many CX leaders still aren’t sure what it means, or whether they actually need it. In theory, it refers to AI that can take initiative, make decisions, and act autonomously.

For now, most CX organizations are focused on more immediate, proven AI use cases that drive real impact.

Hype vs. Reality: Automation Has Limits

While automation was everywhere, a lot of companies think that over-automating means losing the human touch. Over-automation risks damaging customer trust and experience.

Many CX leaders are taking a balanced approach: using automation where it drives efficiency but keeping humans in the loop where empathy matters.

Personal Perspective

The conversations at CCW Vegas reinforced our belief that the future of CX is powered by voice-first, human-centered AI. We heard strong demand for solutions that enhance, not replace, the human element in customer interactions.

It was clear that innovations like Krisp’s real-time Voice AI platform are helping companies meet that need—boosting both agent and customer experience across high-stakes voice channels.

Krisp Milestones

At CCW Vegas, we unveiled our advanced real-time Voice AI platform complete with Accent Conversion v3.5 support for India, the Philippines and Latin America, AI Live Interpreter, Agent Assist (Supervisor Assist coming soon!), and our award-winning noise cancellation technology.

Krisp’s AI Live Interpreter earned runner-up for Disruptive Technology of the Year, recognized for delivering true bidirectional speech-to-speech translation across over 80 languages, well beyond any alternative in the market today.

We also announced a new strategic partnership with Everise, deploying Accent Conversion and AI-powered bidirectional noise and voice cancellation to over 10,000 seats globally, with rapid expansion underway. AI Live Interpreter and Agent Assist are also planned for rollout as part of the partnership.

"Krisp's technology has consistently outperformed in head-to-head evaluations across clarity, naturalness, and accent accuracy." — Sudhir Agarwal, Founder and CEO of Everise

Finally, we were proud to host this year’s CCW Excellence Awards. A huge thanks to our customers and partners who visited our booth and made the week a success.

What’s Next

Looking ahead, the industry’s sharpening its focus on purposeful AI and solutions that clearly improve human outcomes, both for agents and customers.

Voice AI continues to stand out as a high-impact area. As AI maturity evolves, expect to see more thoughtful and orchestrated human-AI experiences across the industry.

Stay tuned for the upcoming Future of Voice AI podcast CCW series, featuring industry experts shaping the next wave of CX innovation.

I thought it was time to revisit the scores.

• “Business Pain” represents how important the problem is for the customer. The higher the pain, the more urgent it is for the customer to solve it.

• “AI Readiness” represents the industry’s technological readiness to solve the pain.

Below is my Q2 2025 version.

Here is what changed in 1.5 years:

• AI Note-Taking went up by 3 points! 🔥🔥🔥

• AI Voice Translation went up by 3 points! 🔥🔥🔥

• AI Accent Conversion went up by 2 points! 🔥🔥

• AI Voice Agents went up by 1 point! 🔥

• AI Live Assist/Guidance went down by 1 point 😞

• I’ve removed AI Voice Conversion altogether 🤔

Let’s go over the pains one by one.

Subscribe now

1) Conversations visibility at scale

You have 100 sales reps (or call center agents) and they receive/place calls all the time trying to close deals or serve customers.

How do you know if they are doing a good job? Just a couple of years ago, this was quite difficult to do. These days, all you need to have is a Conversational Intelligence tool (aka Speech Analytics) and you will get full visibility into every conversation.

Every customer can be recorded, transcribed and summarized for you and the team. These tools are super convenient and there are plenty of them available in the market - Cresta, Observe, Gong, Avoma, CallMiner, SalesLoft, etc.

AI Readiness: 10/10, Pain: 10/10

2) Language barrier

The ability to communicate verbally in an effective way is a foundational capability of any team and business. The language barrier has always been one of the top problems for humanity. It takes years for people to learn to speak in a non-native language. The pain is real and any business that has this pain would pay a lot of money to eliminate it.

Imagine there being a real-time speech-to-speech translation AI that people could use for communication over Zoom, Teams or Krisp. This would be a game-changer.

Krisp, OneMeta, MS Teams, Google Meet already have solutions for this.

AI Readiness: 6/10, Pain: 10/10

3) Conversations at scale

There are multiple roles where the person needs to receive or place calls and talk to another human being on the other side. Call center agents, sales and business development reps, recruiters, and others.

Maintaining and growing such teams is exceptionally difficult. Businesses need to recruit talent, and then they need to onboard and retain them.

This is an expensive endeavor. No doubt, Voice AI Agents are taking over and automate some of these functions.

The space is booming. There are plenty of startups in this space.

AI Readiness: 6/10, Pain: 10/10

4) Taking meeting notes

In many companies, there are people dedicated to taking notes in meetings. For many years this has been a manual task that can be automated with Voice AI now.

Many tools already offer meeting transcription, summary, and follow-up generation.

The quality varies from 60%-80% for now. No doubt it will keep improving and the manual work will be fully automated in the coming year or two.

There are already multiple companies doing this:

• Krisp, Fireflies, Otter, tl;dv, Fathom, Avoma and others

AI Readiness: 7/10, Pain: 8/10

5) Onboarding and training of associates

Call center agent turnover rate is between 30%-45%. This means that a huge number of agents leave every year and managers need to find replacements and onboard/train them. This is a costly process and any automation/simplification of the process has a clear ROI.

Similarly, companies that need to hire a high number of SDRs or AEs, need to onboard and train them, otherwise, their sales conversion rates would decrease. Again, high-ROI endeavor.

Imagine a bot sitting on an agent’s machine that listens to the customer conversation and gives real-time hints that have a history of better conversion rates or customer satisfaction. The agent ramps up quicker due to this technology.

This technology already exists and is called AI Live Assist. Multiple companies already have shipped products with such technology:

• Balto, Observe, Google, Aigent, Five9, Uniphore, LevelAI and others

AI Readiness: 6/10, Pain: 7/10

6) Accent barrier

As with the language barrier, human accent is a serious barrier that impacts understanding and comprehension of business conversations. Nearly all humans have accents when speaking in non-native languages and it’s extremely difficult to retrain them.

Call centers have special training programs for their agents to reduce accent. The cognitive load and stress on agents for such tasks are intense.

Imagine a Voice AI technology that would, in real-time, localize the speaker’s accent to the listener’s accent to improve understanding and comprehension.

Krisp and Sanas have already deployed such technology in the call center industry.

AI Readiness: 8/10, Pain: 7/10

7) Background noises & voices

The problem of background noises and voices in calls has been around for more than 30 years. It creates a distraction for the call participants and prevents them from focusing on the core conversation. Background noise also creates a constant stress for the speakers.

In call centers, background noise can result in a customer satisfaction drop, longer conversations and mental stress for agents.

Luckily, AI-powered Noise Cancellation technology can fully solve this problem. Krisp has pioneered this technology in the industry and has large-scale deployments of it. It solves both the problem of noises as well as background voices. Zoom, MS teams and other applications also have invested in such technologies.

AI Readiness: 10/10, Pain: 6/10

The technical term for this is turn-taking, or interrupt-handling.

Turn-Taking

Turn-taking is a hot problem these days, with many companies trying to solve it.

• LiveKit has published an article showing how they tackle it

• Daily recently started an open-source project called smart-turn

• OpenAI recently launched a variant of it called “semantic VAD”.

A particular case where turn-taking fails miserably is in noisy environments.

Whenever there is background noise or chatter, AI agents get confused and start to interrupt us at the wrong time, talk over us, or miss what’s actually being said. This makes conversations frustrating and unnatural.

In a normal conversation, humans naturally know when to pause, respond, or wait their turn. AI agents don’t have that instinct. Today, they have to rely on Voice Activity Detection (VAD)—a technology that decides when a piece of audio is human speech or not. An AI agent looks into VAD’s output and if there is enough “non-speech” data, they decide the person finished speaking.

However VAD-based turn-detection is too primitive for real-life scenarios. There are many situations where people could pause without finishing their speech.

Here is a great deep-dive video on turn-taking.

Improving turn-taking with Noise and Voice Cancellation

While solving turn-taking is difficult, we can improve it with noise and voice cancellation technology by placing it just before VAD and speech recognition models.

By filtering out background noise and voices in real time, AI agents get only the speech that matters. That means:

• No more false interruptions

• No more missed responses

• Smoother, more human-like conversations

Here’s how a voice agent performs with Daily, Pipecat, and Gemini in a noisy environment—with vs. without noise cancellation:

The Results

Real-world tests show that when background voice and noise cancellation are applied before VAD, AI agents perform much better:

• 3.5x fewer false interruptions → A 71% decrease in AI cutting off users unnecessarily.

• 2x better speech recognition accuracy → AI agents hear and respond more accurately.

• 50% decrease in call drops → Less conversations abandoned due to frustrating interruptions.

• 30% increase in CSAT → Smoother interactions make happier customers.

Leading Conversational AI platforms—including Vodex, Fixie, Daily, LiveKit, and Fluidworks—have already integrated noise and voice cancellation to fix turn-taking and improve response accuracy.

Below is a technical report on how exactly this works:

What This Means for AI Teams

If you’re building or deploying AI voice agents, this is a must-have. Without noise cancellation, AI models are guessing when to talk and when to listen, which leads to broken conversations.

Clean audio means better AI decisions. And better AI decisions mean better user experiences. It’s that simple.

Language barriers have always posed challenges in contact centers. Traditionally, companies relied on human interpreters, but this approach is expensive, slow, and inefficient.

Now, AI technology is transforming how businesses deliver multilingual support, offering scalable, cost-effective, and practical solutions.

The History and Challenges of Human Interpreters

Human interpreters have been the go-to solution for enabling communication across languages. This involves hiring multilingual staff or outsourcing to over-the-phone interpretation (OPI) services. While these methods work, they come with drawbacks:

• High Costs: Multilingual agents and OPI services are expensive, especially in high-volume environments.

• Delays: Connecting to human interpreters adds time, increasing handle times, frustrating customers, and compromising CSAT.

• Scalability: Managing demand surges during peak times is resource-intensive.

• Security Risks: Third-party interpreters raise concerns about data privacy and compliance with sensitive customer information.

Human interpreters are invaluable in some scenarios, but the need for more scalable and efficient solutions is undeniable. The good news is, these solutions are here.

AI Interpretation is Revolutionizing Contact Centers

AI-powered tools are transforming multilingual support by addressing the limitations of human interpreters. Built on advancements in translation AI and GenAI technologies, these systems provide reliable, real-time interpretation.

Introducing Krisp’s AI Live Interpreter

Krisp recently launched AI Live Interpreter, the industry’s first AI live interpretation solution, offering real-time, bi-directional translation. With enterprise-grade scalability and security-first design, it helps call centers eliminate language barriers with the click of a button.

Key benefits:

• Instant Availability: It’s available 24/7, eliminating delays.

• Cost Efficiency: AI Live Interpreter operates at a fraction of the cost of human interpreters.

• Scalability: Works with all softphones out-of-the-box, and is built on systems designed to handle unlimited simultaneous sessions, adapting effortlessly to peak demand.

• User Experience: Agents sees both live transcription as well translation in front of them which helps to get more context

• Security: Many AI solutions are privacy-first, reducing risks associated with third-party interpreters.

Krisp supports over 25 languages with high quality and counting.

Let’s see it in action.

English + Mexican Spanish language pairing:

English + Hindi language pairing:

See it live and learn how Krisp’s AI Live Interpreter can transform your global support operations.

Book a demo

BLEU: Setting Standards for Translation Quality

Ensuring translation accuracy is crucial for customer interactions. The Bilingual Evaluation Understudy (BLEU) methodology is a widely recognized metric for measuring the quality of translations—both human and AI-generated. It compares machine-generated translations to human references and assigns a score based on similarity.

Here’s how BLEU scores are interpreted:

• Human Translations: Typically score around 60, reflecting their flexibility and nuance.

• AI Translation: Scores of 20-30 are considered sufficient for effective communication, with scores above 30 indicating high-quality outputs.

• Short Phrases (~5 seconds): AI translations scoring above 40 are nearly indistinguishable from human translations.

• Krisp’s: Sees Scores averaging 30-45 across supported languages, ensuring robust performance for real-world customer interactions.

Although AI systems may not match human-level nuance, their BLEU scores demonstrate they are highly capable for contact center needs. Businesses can rely on these solutions for clear, consistent, and accurate communication in real time.

Democratizing Multilingual Support

AI interpretation technology is leveling the playing field for businesses of all sizes, enabling them to offer multilingual support without massive overhead:

• For Small Businesses: Affordable, easy-to-deploy solutions help smaller companies compete globally.

• In Emerging Markets: Companies in regions with diverse languages can scale quickly without hiring large multilingual teams.

By removing cost and scalability barriers, AI is making multilingual customer support a practical standard rather than a costly exception.

Challenges and the Road Ahead

Despite its advantages, AI interpretation still has limitations:

• Accuracy: Industry-specific jargon and heavy accents can impact performance.

• Cultural Nuance: Machines may struggle with subtle emotional or cultural cues.

• Integration: Ensuring AI tools align seamlessly with workflows requires thoughtful implementation.

However, as machine learning models continue to improve, AI tools are becoming more adaptable and precise. The use of BLEU scores validates their progress and readiness for real-world applications.

Looking Ahead

The future of customer support lies in AI-powered tools. Businesses that embrace these technologies early can streamline operations, reduce costs, and enhance customer satisfaction. AI interpretation is no longer a luxury but a necessity in today’s interconnected market.

AI is transforming contact centers, making multilingual support a reality for everyone. For companies looking to stay ahead, now is the time to act.

Learn more

The Realtime API uses both text tokens and audio tokens. 
Text input tokens are priced at $5 per 1M and $20 per 1M output tokens. Audio input is priced at $100 per 1M tokens and output is $200 per 1M tokens. 
This equates to approximately $0.06 per minute of audio input and $0.24 per minute of audio output.

This translates to $0.15 per minute. It’s arguably quite expensive and higher than expected, leaving many wondering how it compares to the cost of human agents.

This pricing is less expensive than the hourly rate of human agents in the U.S. and U.K., but 2x the cost of human agents in the Philippines and 5x the cost of those in India.

What does this mean for customer support?

There’s been a lot of talk about what Voice mode means for the industry.

There are two main opinions on its impact:

• Bot enthusiasts: People who boast AI bots will very soon completely replace human agents. Likely affiliated with voice bot companies.

• Traditionalists: Legacy thinkers who are skeptical of AI’s potential to replace agents due to the complexity of the industry and technology’s immaturity.

The truth, as almost always, falls somewhere in the middle—AI will eventually disrupt the industry entirely, however it will take way more time.

Let’s get some things straight

• The contact center industry is huge and complex

• Major changes will not happen overnight

• Customer acceptance will be the ultimate driver in bot deployment and adoption

• Millions of people losing jobs is a major political problem. If the process starts to get regulated, it may take much longer.

Voice Bots: Pros and Cons

• Pros

  • The price of Voice AI will go down by 2-3x in a year

  • Voice bots will cover more and more use cases

  • Much easier to manage bots than people

  • Bots work 24/7, without breaks and don’t get tired

  • No paying overtime

  • Speaks multiple languages

  • No onboarding and training

  • Easy to scale.

• Cons:

  • AI hallucinates and, until this is solved, AI won’t be trusted

  • Hallucination errors add up and compound when multiple agentic flows are bound together

  • Large integration cost with legacy systems

Where Humans Outperform

• Show empathy, read between the lines, and connect with customers

• Have intuition, can adapt, and are better at handling complex or sensitive issues

• Create trust and rapport with customers

What happens next?

Few things should happen for the adoption of Bots to accelerate

• Must stop hallucinating

• Reasoning must get better

• The price must go down

• Must have more integrations into enterprise systems and tooling

Subscribe now

Voice AI Newsletter

Voice channel is and will stay #1 in CX

For years, we’ve heard that the voice channel (phone/live agents) in contact centers is fading into obsolescence. Yet despite the rapid growth of digital platforms, customers continue to seek human interaction for support…

Read more

2 years ago · 2 likes · Davit Baghdasaryan

CCW reports that over 80% of customers still prefer voice-based human support. Customer service needs to be quick and efficient.

• 64.4% of inbound interactions are voice calls (live agent) 🔥 Whopping 84% in the public sector 😲

• 89% of customers want the reassurance of talking to a person

• 25% CX leaders think the voice channel will "greatly increase" in 2024 📈

• 35% CX leaders will invest more in the Voice channel in 2024

However long wait times, poor quality, language barriers and repetitive questions frustrate both customers and agents. Modernizing CX with voice AI is crucial for solving these pain points.

Challenges today

Though customers prefer voice over other channels, it’s not without its flaws. There are three main pain points:

• Long wait times

• Repetitions

• Poor audio quality

• Poorly trained agents

• Accents and language barriers

These problems lead to misunderstandings, repeated information, and customer dissatisfaction. For businesses, this means more time spent on each call and higher operational costs.

So why does voice still win?

Even though voice has multiple challenges, customers still prefer it when the problem they have meets one of the following criteria:

• Complex, OR

• Emotional and Complex, OR

• Emotional and Complex and Urgent

Voice AI will fix all this

I have no doubt, Voice AI will solve all these challenges.

I strongly believe that the impact of Voice AI will be very significant on voice channel.

We’ll dive into all of this live at Krisp Fullband 2024 on Sep 25th.

Tune in to unpack 10 months of insights from CX leaders, groundbreaking research and more.

Subscribe now

The study surveyed BPOs with offshore operations and U.S. consumers, revealing important insights into how accent-based misunderstandings affect both customer experience and agent well-being.

Key Findings:

• Customer Experience Discrepancy: There’s a significant gap between what contact centers believe and what customers actually want. Only 16% of contact centers thought that U.S.-based agents were a top priority for customers but 35% of U.S. consumers say it is a key factor in their customer experience. This shows that many BPOs do not fully understand their customers' needs.

  

• Impact on Hiring: Every BPO in the study admitted that an agent's accent plays a crucial role in hiring decisions. On average, 64% of potential agents are not hired because of their accent, rising even more in South Africa and India.

  

• Repetitions and Costs: Accent-related misunderstandings lead to frequent repetition requests. About 50% of U.S. consumers feel uncomfortable asking offshore agents to repeat themselves, yet 80% of calls involve some form of repetition. These misunderstandings frustrate customers and increase call times, costing the industry over $5.5 billion annually.

  

This study highlights the need for BPOs to better understand and address the challenges posed by accents in offshore contact centers.

By recognizing these issues and considering voice AI technology, like AI Accent Localization and AI Interpreter, BPOs can improve both customer satisfaction and operational efficiency.

📊Access the full report “Can you repeat that?” The $5.5 Billion Impact of Language Barriers on Offshore Agents & Contact Centers

Read the Report

Also, on Sep 25th join me and Steve Morrell from ContactBabel as we discuss more key findings from the report, live at Fullband 2024 🔥

Register now

Sources:

1.  ContactBabel, “2024 US Contact Center Decision Makers’ Guide”

2.  ContactBabel, “US Contact Centers 2023-2027: The State of the Industry”

3.  Helpware, “Global Adoption of Offshore Outsourcing: Statistics”

I wrote about it here:

Voice AI Newsletter

The AI PC Era 🚀 for Call Centers is here

“The AI PC will be a sea change moment in technical innovation…

Read more

2 years ago · 5 likes

And here:

Voice AI Newsletter

The Power of On-Device Transcription in Call Centers 💪

With advancements in Speech-to-text AI and on-device AI, the call center industry is approaching a transformative change. We should start rethinking the traditional approach of cloud-based transcriptions, bringing the process directly onto the agents' devices…

Read more

2 years ago · Davit Baghdasaryan

This trend allows more Speech-to-Text workloads to be moved to on-device AI.

This is incredibly valuable in call centers and BPOs.

The problem with STT is that you need to first obtain the recordings from SoftPhone platforms and unfortunately it's not always possible.

- There might be no integration available with Softphone

- The cost might be prohibitive

- Customer might not allow it due to compliance

- It might take days to obtain it

Introducing Krisp Speech-to-Text API 🔥

At Krisp, we just launched an on-device Speech-to-Text API, specifically designed for call centers and BPOs.

• Automatically supports all voice platforms (no integration needed)

• Up to 10x cheaper than the industry

• Customer data is not sent to any servers (including Krisp's)

• Both post-call and real-time

• Even PII/PCI is removed on-device

Who is it built for?

We have been building this technology for >2 years and are excited to bring it to our call center customers.

The solution is perfect for call centers and BPOs wanting to build Speech Analytics, Customer QA or Agent Assist technologies to make their operations more effective. Speech-to-Text is a core building block for these technologies.

It’s also perfect for enterprises that don’t want to share internal or customer data with 3rd parties.

Since Speech-to-Text happens on-device, the pricing is disruptive.

How it works

The installation process is straightforward:

1. Install Krisp app on agents’ devices (the same app that provides Noise Cancellation and Accent Localization)

2. Turn on Speech-to-Text from Krisp’s web admin dashboard

3. Specify the private cloud where you want call transcripts to be uploaded (e.g. S3)

Once set up, as soon as a call ends, Krisp will upload the transcript to the private cloud location, with < 1 second latency.

If Recording is enabled, Krisp will also record calls and upload recordings to the same private cloud, along with call transcripts.

Integration with softphone systems

The solution automatically integrates with top CX and voice platforms such as Genesys, Avaya, TalkDesk, Teams, Zoom, and more, simplifying the implementation process. No integration is required.

Accuracy

Krisp-generated transcripts go through several post-processing steps to make sure the transcript has the highest quality level:

• Accuracy with a WER (Word Error Rate) of only 7.96%

• Adds punctuation, capitalization, and numerical values

• Assigns text to speakers with timestamps

• If enabled, removes PII/PCI and filler words

Supporting 4 languages

Krisp’s STT API currently supports English, German, French and Spanish. More languages will come over time.

Learn More

You can learn more here.

From what we’ve heard, it will be even bigger next year. CCW will expect 5000 attendees in 2025.

Krisp’s booth at the Expo:

There were 2 Expo halls this year and a lot of companies showcasing their products.

Summary of the companies exhibiting:

• A lot of companies showcasing AI Voice Agent products

• A lot of companies showcasing AI Agent Assist products

• 3 companies showcasing AI Accent Localization

• (at least) 2 companies showcasing AI Voice Translation

It’s becoming clear that there are two camps of Voice AI products in CX:

• Agent Assists - products that help agents to be more productive

• AI Voice Agents - products that build autonomous voice-based agents

I sat down with 5 amazing executives and thought leaders onsite to discuss the Future of Voice AI:

• Eric Guarro, SVP of Digital Transformation, ibex

• Natalie Beckerman, Global Head of Customer Support Operations, IHG

• Shep Hyken, Customer Experience Expert | NYT Bestselling Author

• Philip Bennett, Head of Innovation, Empire Today

• Praveer Chadha, SVP Customer Management, Datamatics

Here is the recap of the interviews.

I will be publishing these insightful interviews in a 4-week CCW series.

In our discussions, key themes emerged: integrating AI, training AI Agents in emotional intelligence, scaling multi-language support for global growth, and more. I'll explore these topics in depth throughout the series.

One thing everyone seems to be aligned on is the narrative that

• AI Agents will take over easy, routine tasks while frontline workers will focus on more complicated customer interactions and escalations.

• Powerful and seamless Agent Assist products will enable human agents to provide deeper, more complex support, enhancing their productivity and effectiveness.

And both are super important categories for the future of customer experience.

This synergy promises to elevate customer satisfaction and operational efficiency across the industry.

Stay tuned!

Subscribe now

Subscribe now

What is On-Device AI?

On-Device AI refers to the deployment of AI models directly on-device. Unlike Cloud AI, which relies on centralized cloud-based systems for processing and decision-making, On-Device AI processes data locally on the device. This shift in the computing paradigm offers numerous advantages for contact centers aiming to improve customer service.

Benefits of On-Device AI in Contact Centers

1. Reduced Latency

One of the most significant advantages of on-device AI is its ability to reduce latency. In a contact center environment, every millisecond counts. Cloud-based technology requires data to be sent to a remote server for processing, which can introduce delays. AI that runs on-device eliminates this round-trip by processing data locally, resulting in near-instantaneous responses.

For instance, AI-powered speech to text technology operating on-device can understand and process spoken words in real-time, without the lag associated with cloud processing. This leads to more accessible transcriptions.

2. Improved Real-Time Decision-Making

Contact centers often deal with a high volume of customer interactions, each requiring quick and accurate decisions. On-Device AI enables real-time data processing and analysis, empowering agents with actionable insights at the moment of interaction. This can include everything from sentiment analysis and customer behavior prediction to real-time translation and personalized recommendations.

By bringing intelligence closer to the customer interaction, contact centers can make more informed decisions faster, ultimately enhancing the overall customer experience.

3. Cost Efficiency

Processing data in the cloud can be expensive, especially when dealing with large volumes of data and high-frequency interactions. On-Device AI reduces the need for constant data transmission to and from the cloud, resulting in lower bandwidth costs and reduced cloud service fees.

Additionally, devices can handle many AI tasks independently, reducing the load on central servers and allowing for more efficient resource allocation. This cost efficiency can be a game-changer for contact centers looking to optimize their operations without compromising on performance.

4. Enhanced Security and Privacy

Data security and privacy are paramount in contact centers, where sensitive customer information is often handled. On-Device AI enhances security by keeping data processing local, minimizing the exposure of sensitive information to potential breaches during transmission.

Real-World Applications of On-Device Voice AI in Contact Centers

AI Accent Localization

AI Accent Localization is a compelling application of on-device AI in contact centers, enabling the understanding and processing of diverse accents in real-time. This technology ensures that customer interactions are smooth and efficient, regardless of the customer's native accent or regional dialect. By operating directly on the device, Accent Localization provides quick, accurate responses without the delays associated with cloud processing.

Real-Time Transcription and Sentiment Analysis

Understanding customer sentiment is crucial for delivering personalized and empathetic service. On-device transcription and AI can analyze customer sentiment in real-time during interactions, enabling agents to adjust their approach based on the customer's emotional state. This leads to more positive and meaningful customer experiences.

Real-time Noise Cancellation

On-device AI-powered noise cancellation significantly enhances the clarity of customer interactions by filtering out background noise in real-time. This technology ensures that both customers and agents communicate more effectively, leading to higher quality conversations and improved customer satisfaction. In high-volume environments like contact centers, this capability drastically reduces misunderstandings and the need for repetition, streamlining operations and enhancing overall efficiency.

Fraud Detection and Prevention

Enhancing security measures by detecting fraudulent activities in real-time, AI can analyze voice patterns during a call to identify potential fraudsters, alerting agents to take necessary precautions. This proactive approach helps protect both the contact center and its customers from malicious activities.

Conclusion

As contact centers strive to meet the evolving demands of customers, On-Device AI emerges as a powerful tool for reducing latency, improving real-time decision-making, and enhancing overall customer experience. By processing data locally and bringing intelligence closer to customer interactions, On-Device AI offers numerous benefits, including cost efficiency, enhanced security, and simplified management.

The demo app was ChatGPT and the demo’s focus was their new Voice mode.

The demos were exceptional and quite futuristic!

OpenAI’s engineering team has figured out a way to map audio to audio directly as a first-class modality which reduced the latency and added more “audio intelligence” to the model.

The result is a low-latency and natural-sounding conversational AI.

Many startups have been trying to do this for a while but bringing the latency down was a challenge.

It turns out that having an end-to-end trained speech foundational model was the solution.

The beauty of this model is that it is able to perform many tasks in parallel:

• Transcribe (even better than Whisper)

• Translate (better than many existing models)

• Reason better than GPT-4.5 and other models

• Generate fast response

So how will this impact Voice Bots (e.g. in Call Centers)?

Once GPT-4o voice mode is made available, the companies will switch to it. Their voice bots will:

• sound more natural

• will have 2-3x lower latency

• will speak different languages

The adoption of Voice Bots products will simply accelerate. Exciting times ahead!

• AI bots will have a transformational impact on Call Center BPOs

• More frontline workers will switch to solving L2 and L3 customer problems instead of L1

• BPOs will need to empower their workers with AI and create “SuperAgents”

• Transcription technology is a key infrastructure for SuperAgents however transcription is quite expensive. Hence cheap transcription technology is a critical component.

• On-device transcription running on the agent’s device could be the perfect solution to this. There are already providers delivering such products.

AI Voice Bots and BPOs

After ChatGPT, all BPO call center companies are trying to understand how AI is going to impact the future of their companies.

Call center BPOs (Business Process Outsourcing) are companies that manage customer interactions on behalf of other businesses. They handle a wide range of services, including customer support, technical assistance, sales, and other client-related functions. These centers can operate across various communication channels like phone, email, chat, and social media. 

BPOs often provide cost-effective solutions for businesses by handling large volumes of calls and communications, allowing companies to focus on their core activities while maintaining quality customer service. They can operate domestically or internationally, leveraging global talent to provide 24/7 support and services in multiple languages.

The number of AI Voice Bot startups attempting to replace frontline workers (agents) with AI is growing significantly. These are some of the hottest startups attracting VC funding these days. The promise is that businesses can support their customers entirely with AI, reducing operational costs and increasing customer satisfaction.

One might think that AI is simply going to “eat” customer support and call centers, however of course things are more nuanced, as everything else in life.

Customer Ticket Levels

First, we need to understand that customer support doesn’t come in one shape. Typically there are 4 levels of tickets, with varying complexity.

• L0: Self-service
  No direct support; customers use FAQs and automated tools.

• L1: Basic Support
  Handles common, simple issues like password resets.

• L2: Technical Support
  Deals with more complex problems requiring technical skills.

• L3: Expert Support
  Addresses the most complex issues, often involving system changes or code fixes.

L0 tickets are already handled by automated tools like chatbots and helpdesk.

It’s quite clear that LLM-powered AI is going to have large impact on L1 tickets. In the next 2-5 years AI-powered voice and chat bots are going to handle the majority of such tickets. In other words, the scope of L0 is going to expand into L1

The number of frontline workers focused on L1 tickets will obviously decrease.

However, when it comes to more technically challenging tickets (L2 and L3), AI’s impact is not clear yet. LLMs are not smart enough to be trusted to autonomously solve complex technical problems.

More Workers in L2 and L3

As AI takes over L1 tickets, companies will save money.

With more budget in hand, they will want to invest in improving the support of L2 and L3 tickets. After all, most companies always strive for better customer satisfaction.

Re-qualifying people from L1 to L2 and hiring new people to handle L2/L3 tickets will become a trend.

This is one area where BPOs will continue provide value to businesses - qualified workers handling more difficult tasks than AI can do.

Enter SuperAgents

Another trend that will indeed happen is the introduction of ‘Super Agents’ - frontline workers equipped with amazing AI tools that empower them to be more productive.

Frontline workers will use CoPilots before, during and post customer interactions.

These technologies are already being deployed in call centers and BPOs and are already showing great results in reducing waiting queues, handling time and increasing CSAT.

Examples:

• AI Noise Cancellation

• AI Accent Localization

• AI Speech-to-Speech Translation

• Chat-based CoPilot for Agents integrated with knowledge base

• Voice-based CoPilot for Agents integrated with knowledge base

• CoPilots for extracting insights from product logs

• CoPilots for suggesting the best possible responses on calls

Today, many large BPOs are already integrating (or building) such technologies, supporting the vision of SuperAgents.

Transcription is Critical

One of the foundational technologies to enable super agents in BPOs is Speech-to-Text (aka Transcription).

Many modern BPOs still struggle getting access to conversations from their thousands of frontline workers. There are various reasons for this:

• Cloud-based Speech-to-Text is expensive

• BPO customers don’t want to share transcriptions with BPOs

• Even if they share, it might take weeks to get these

• BPOs operate in many locations, with agents using different CCaaS stacks, so getting transcriptions is consistently difficult

We have talked before about 2 new global trends solving these challenges

• On-Device Transcription

• AI PC Era

Below is a demo of how On-Device Transcription works for Call Centers.

These two trends will enable BPOs to easily fill the gap in transcriptions and build powerful CoPilots for future Super Agents.

This guide, created in partnership with ContactBabel, contains comprehensive research, actionable insights and guidance to help contact centers navigate the future of voice communication in the call center industry, where clarity, comprehension, and connection redefine customer interactions.

The Effect of Audio Improvement on Productivity and CX

In a ContactBabel survey of 1,000 customers, 60% of customers frequently face challenges in understanding agents due to poor audio quality. 

Some businesses experience higher levels of audio interference due to their contact center environment, use of remote working, and type of customer (e.g. older customers experience this the most). Those taking calls from customers on mobile phones are more likely to have higher rates of repetition. 

Lack of audio clarity is not just a problem on the contact center’s side of the conversation. With more people than ever using mobile phones to speak with organizations, both agents and customers have to concentrate very hard on the conversation, causing stress and frustration, particularly for the agent who may handle 80-100 calls each day.

Real-world example

A Spanish contact center gave some sets of headsets with digital audio processors to employees, while others used the more traditional headset. The first group's technology had the effect of 'cleaning up' unwanted noise at either end of the line, allowing the customer and employee to communicate more effectively. Calls were handled more quickly, fewer mistakes were made with data collection (with the attendant knock-on effect that fewer repeat calls were required), and overall, employees handled an average of 10% more calls per day compared to the control group. 

AI-enabled voice isolation can intelligently remove background noise from both sides of the conversation, in real-time, to assist the smooth and accurate flow of the conversation, as well as in recordings to improve post-call analytics and voice-to-text transcription. This also means that businesses spend significantly less on upgrading and replacing top-of-the-line headsets. 

Noise Canceling Headsets and Technology

61% of surveyed individuals indicate their headsets are equipped with noise-canceling microphones to reduce background noise, enhancing call clarity for the caller and reducing the need for repetition.

However, only 37% have noise-canceling headphones for all headsets, leaving a significant number of agents exposed to noisy environments. And these headsets don’t address inbound noise coming from the customer’s environment. The implication is reduced focus, accuracy, and overall performance, potentially prolonging calls. 

Figure 2: Use of noise-canceling microphones and headphones/earphones

Alternatively, noise-canceling software offers a higher quality, more accessible and scalable solution compared to traditional noise-canceling headsets. Because this technology works directly on agent devices and can be centrally managed, implementation and adoption are easy and streamlined for call centers of all sizes.

Unlike headsets that require individual distribution and maintenance and hardware refresh, AI-powered noise cancellation technology can be updated and improved from one central point, ensuring the on-device technology is always up-to-date and active, making high-quality audio more achievable and consistent.

Improving call center audio quality and eliminating noise with Voice AI

AI-powered voice technology is revolutionizing the way call centers manage voice interactions and quality. By filtering out background noise, echoes, and other human voices, Voice AI ensures that the primary message is heard, understood, and improves the overall customer experience and satisfaction.

Voice AI technology cleanses both inbound and outbound audio streams, ensuring clear, concise communication. This breakthrough not only improves the quality of each call but also boosts the efficiency and effectiveness of call center operations, which has massive impact at scale—reducing the $1.34B cost of poor audio quality for contact centers, industry wide. 

‘I’m sorry, can you repeat that?’

Reducing the number of times an agent or customer has to repeat themselves can make a huge impact on call center costs by reducing call times (and thus queue lengths)  while also improving the overall customer experience. 

Call clarity directly translates to improved first call resolution rates. Issues are resolved faster, and the need for callbacks or escalations are significantly reduced. This efficiency is a win-win: customers experience improved satisfaction, and contact centers benefit from reduced operational costs and experience 10% increases in agent productivity, as seen in the example referenced earlier.

The reduction in noise complaints speaks volumes—literally.

With up to a 78% decrease in noise-related issues, Voice AI is improving call productivity, while eliminating agent stress and customer frustration.

By effectively eliminating background noise and improving audio quality, AI Noise Cancellation reduces the need for expensive headsets and physical modifications like soundproof partitions or white noise machines.

Replacing traditional, expensive, noise cancellation solutions

The shift from hardware to AI Noise Cancellation software lowers initial investment costs while decreasing ongoing maintenance expenses and providing scalability without compromising quality or incurring additional overhead. As a result, contact centers can allocate their resources more efficiently, investing in areas that directly contribute to growth, customer satisfaction, and scaling their operations. 

This article was originally posted on Krisp AI blog.

In this article, we dive deep into a new disruptive technology called AI Accent Localization, which in real-time translates a speaker’s accent to the listener’s natively understood accent, using AI.

Accent refers to the distinctive way in which a group of people pronounce words, influenced by their region, country, or social background. In broad terms, English accents can be categorized into major groups such as British, American, Australian, South African, and Indian among others. 

Accents can often be a barrier to communication, affecting the clarity and comprehension of speech. Differences in pronunciation, intonation, and rhythm can lead to misunderstandings. 

While the importance of this topic goes beyond call centers, our primary focus is this industry.

Accented speech in call centers

The call center industry in the United States has experienced substantial growth, with a noticeable surge in the creation of new jobs from 2020-onward, both on-shore and globally.

In 2021, many US based call centers expanded their footprints thanks to the pandemic-fueled adoption of remote work, but growth slowed substantially in 2022. Inflated salaries and limited resources drove call centers to deepen their offshore operations, both in existing and new geographies.

There are several strong incentives for businesses to expand call centers operations to off-shore locations, including:

Cost savings: Labor costs in offshore locations such as India, the Philippines, and Eastern Europe are up to 70% lower than in the United States.

Access to diverse talent pools: Offshoring enables access to a diverse talent pool, often with multilingual capabilities, facilitating a more comprehensive customer support service.

24/7 coverage: Time zone differences allow for 24/7 coverage, enhancing operational continuity.

However, offshore operations come with a cost. One major challenge offshore call centers face is decreased language comprehension. Accents, varying fluency levels, cultural nuances and inherent biases lead to misunderstandings and frustration among customers. 

According to Reuters, as many as 65% of customers have cited difficulties in understanding offshore agents due to language-related issues. Over a third of consumers say working with US-based agents is most important to them when contacting an organization.

Accents create challenges in call centers

While the world celebrates global and diverse workforces at large, research shows that misalignment of native language backgrounds between speakers leads to a lack of comprehension and inefficient communication. 

• Longer calls: Thick accents contribute to comprehension difficulties, causing higher average handle time (AHT) and also lower first call resolutions (FCR).
  According to ContactBabel’s “2024 US Contact Center Decision Maker’s Guide” the cost of mishearing and repetition per year for a 250-seat contact center exceeds $155,000 per year.
  

• Decreased customer satisfaction: Language barriers are among the primary contributors to lower customer satisfaction scores within off-shore call centers. According to ContactBabel, 35% of consumers say working with US-based call center agents is most important to them when contacting an organization.

• High agent attrition rates: Decreased customer satisfaction creates high stress, in turn decreasing agent morale. The result is higher employee turnover rates. In 2023, US contact centers saw an average annual agent attrition rate of 31%, according to The US Contact Center Decision Makers’ Guide to Agent Engagement and Empowerment.

• Increased onboarding costs: The need for specialized training programs to address language and cultural nuances further adds to onboarding costs. 

• Limited talent pool: Finding individuals who meet the required linguistic criteria within the available talent pool is challenging. The competitive demand for specialized language skills leads to increased recruitment costs. 

How do call centers mitigate accent challenges?

Training

Accent neutralization training is used as a solution to improve communication clarity in these environments. Call Centers invest in weeks-long accent neutralization training as part of agent onboarding and ongoing improvement. Depending on  geography, duration, and training method, training costs can run $500-$1500 per agent during onboarding. The effectiveness of these training programs can be limited due to the inherent challenges in altering long-established accent habits. So, call centers may find it necessary to temporarily remove agents from their operational roles for further retraining, incurring additional costs in the process.

Limited geography for expansion

Call centers limit their site selection to regions and countries where accents of the available talent pool is considered to be more neutral to the customer’s native language, sacrificing locations that would be more cost-effective.

Enter AI-Powered Accent Localization

Recent advancements in AI have introduced new accent localization technology. This technology leverages AI to translate source accents to targets accent in real-time, with the click of a button. While the technologies in production don’t support multiple accents in parallel, over time this will be solved as well.

Demo

Below is the evolution of Krisp’s AI Accent Localization technology over the past 2 years.

v0.1 The Very First model (it was bad)

v0.2 A bit more natural sound

v0.3 A bit more natural sound

v0.4 Improved voice

v0.5 Improved intonation

Deploying in the call center

There are various ways AI Accent Localization can be integrated into a call center’s tech stack.

It can be embedded into a call center’s existing CX software (e.g. CCaaS and UCaaS) or installed as a separate application on the agent’s machine (e.g. Krisp).

Currently, there are no CX solutions in market with accent localization capabilities, leaving the latter as the only possible path forward for call centers looking to leverage this technology today.

Applications like Krisp have accent localization built in their offerings. 

These applications are on-device, meaning they sit locally on the agent’s machine. They support all CX software platforms out of the box since they are installed as a virtual microphone and speaker. 

AI runs on an agent’s device so there is no additional load on the network.

The deployment and management can be done remotely, and at scale, from the admin dashboard.

Challenges of building such technology

At a fundamental level, speech can be divided into 4 parts: voice, text, prosody and accent.

Accents can be divided into 4 parts as well – phoneme, intonation, stress and rhythm.

In order to localize or translate an accent, three of these parts must be changed – phoneme pronunciation, intonation, and stress. Doing this in real-time is an extremely difficult technical problem.

While there are numerous technical challenges in building this technology, we will focus on eight majors.

1. Data Collection

2. Speech Synthesis

3. Low Latency

4. Background Noises and Voices

5. Acoustic Conditions

6. Maintaining Correct Intonation

7. Maintaining Speaker’s Voice

8. Wrong Pronunciations

Let’s discuss them individually.

1) Data collection

Collecting accented speech data is a tough process. The data must be highly representative of different dialects spoken in the source language. Also, it should cover various voices, age groups, speaking rates, prosody, and emotion variations. For call centers, it is preferable to have natural conversational speech samples with rich vocabulary targeted for the use case.

There are two options: buy ready data or record and capture the data in-house. In practice, both can be done in parallel.

An ideal dataset would consist of thousands of hours of speech where source accent utterance is mapped to each target accent utterance and aligned with it accurately. 

However, getting precise alignment is exceedingly challenging due to variations in the duration of phoneme pronunciations. Nonetheless, improved alignment accuracy contributes to superior results.

2) Speech synthesis

The speech synthesis part of the model, which is sometimes referred to as the vocoder algorithm in research, should produce a high-quality, natural-sounding speech waveform.  It is expected to sound closer to the target accent, have high intelligibility, be low-latency, convey natural emotions and intonation, be robust against noise and background voices, and be compatible with various acoustic environments.

3) Low latency

As studies by the International Telecommunication Union show (G.114 recommendation), speech transmission maintains acceptable quality during real-time communication if the one-way delay is less than approximately 300 ms. Therefore, the latency of the end-to-end accent localization system should be within that range to ensure it does not impact the quality of real-time conversation.

There are two ways to run this technology: locally or in the cloud. While both have theoretical advantages, in practice, more systems with similar characteristics (e.g. AI-powered noise cancellation, voice conversion, etc.) have been successfully deployed locally. This is mostly due to hard requirements around latency and scale.

To be able to run locally, the end-to-end neural network must be small and highly optimized, which requires significant engineering resources.

4) Background noise and voices

Having a sophisticated noise cancellation system is crucial for this Voice AI technology. Otherwise, the speech synthesizing model will generate unwanted artifacts.

Not only should it eliminate the input background noise but also the input background voices. Any sound that is not the speaker’s voice must be suppressed.

This is especially important in call center environments where multiple agents sit in close proximity to each other, serving multiple customers simultaneously over the phone. 

Detecting and filtering out other human voices is a very difficult problem. As of this writing, to our knowledge, there is only one system doing it properly today – Krisp’s AI Noise Cancellation technology.

5) Acoustic conditions

Acoustic conditions differ for call center agents. The sheer volume of combinations of device microphones and room setups (accountable for room echo) makes it very difficult to design a robust system against such input variations. 

6) Maintaining the speaker’s intonation

Not transferring the speaker’s intonation in the generated speech will result in a robotic speech that sounds worse than the original. 

Krisp addressed this issue by developing an algorithm capturing input speaker’s intonation details in real-time and leveraging this information in the synthesized speech. Solving this challenging problem allowed us to increase the naturalness of the generated speech.  

7) Maintaining the speaker’s voice

It is desirable to maintain the speaker’s vocal characteristics (e.g., formants, timbre) while generating output speech. This is a major challenge and one potential solution is designing the speech synthesis component so that it generates speech conditioned on the input speaker’s voice ‘fingerprint’ – a special vector encoding a unique acoustic representation of an individual’s voice.

8) Wrong pronunciations

Mispronounced words can be difficult to correct in real-time, as the general setup would require separate automatic speech recognition and language modeling blocks, which introduce significant algorithmic delays and fail to meet the low latency criterion.

3 Technical Approaches

Approach 1: Speech → STT → Speech

One approach to accent localization involves applying Speech-to-Text (STT) to the input speech and subsequently utilizing Text-to-Speech (TTS) algorithms to synthesize the target speech. 

This approach is relatively straightforward and involves common technologies like STT and TTS, making it conceptually simple to implement. 

STT and TTS are well-established, with existing solutions and tools readily available. 

Integration into the algorithm can leverage these technologies effectively. These represent the strengths of the method, yet it is not without its drawbacks. There are 3 of them:

• The difficulty of having accent-robust STT with a very low word error rate.

• The TTS algorithm must possess capabilities to manage emotions, intonation, and speaking rate, which should come from original accented input and produce speech that sounds natural.

• Algorithmic delay within the STT plus TTS pipeline may fall short of meeting the demands of real-time communication.

Approach 2: Speech → Phoneme → Speech

First, let’s define what a phoneme is. A phoneme is the smallest unit of sound in a language that can distinguish words from each other. It is an abstract concept used in linguistics to understand how language sounds function to encode meaning. Different languages have different sets of phonemes; the number of phonemes in a language can vary widely, from as few as 11 to over 100. Phonemes themselves do not have inherent meaning but work within the system of a language to create meaningful distinctions between words. For example, the English phonemes /p/ and /b/ differentiate the words “pat” and “bat.”

The objective is to first map the source speech to a phonetic representation, then map the result to the target speech’s phonetic representation (content), and then synthesize the target speech from it.

This approach enables the achievement of comparatively smaller delays than Approach 1. However, it faces the challenge of generating natural-sounding speech output, and reliance solely on phoneme information is insufficient for accurately reconstructing the target speech. To address this issue, the model should also extract additional features such as speaking rate, emotions, loudness, and vocal characteristics. These features should then be integrated with the target speech content to synthesize the target speech based on these attributes.

Approach 3: Speech → Speech

Another approach is to create parallel data using deep learning or digital signal processing techniques. This entails generating a native target-accent sounding output for each accented speech input, maintaining consistent emotions, naturalness, and vocal characteristics, and achieving an ideal frame-by-frame alignment with the input data. 

If high-quality parallel data are available, the accent localization model can be implemented as a single neural network algorithm trained to directly map input accented speech to target native speech.

The biggest challenge of this approach is obtaining high-quality parallel data.The quality of the final model directly depends on the quality of parallel data. 

Another drawback is the lack of integrated explicit control over speech characteristics, such as intonation, voice, or loudness. Without this control, the model may fail to accurately learn these important aspects.

Measuring the speech quality

High-quality output of accent localization technology should:

1. Be intelligible

2. Have little or no accentedness (the degree of deviation from the native accent)

3. Sound natural

To evaluate these quality features, we use the following objective metrics: 

• Word Error Rate (WER)

• Phoneme Error Rate (PER)

• Naturalness prediction

Word Error Rate (WER)

WER is a crucial metric used to assess STT systems’ accuracy. It quantifies the word level errors of predicted transcription compared to a reference transcription. 

To compute WER we use a high-quality STT system on generated speech from test audios that come with predefined transcripts. 

The evaluation process is the following:

• The test set is processed through the candidate accent localization (AL) model to obtain the converted speech samples.

• These converted speech samples are then fed into the STT system to generate the predicted transcriptions.

• WER is calculated using the predicted and the reference texts.

The assumption in this methodology is that a model demonstrating better intelligibility will have a lower WER score.

Phoneme Error Rate (PER)

The AL model may retain some aspects of the original accent in the converted speech, notably in the pronunciation of phonemes. Given that state-of-the-art STT systems are designed to be robust to various accents, they might still achieve low WER scores even when the speech exhibits accented characteristics. 

To identify phonetic mistakes, we employ the Phoneme Error Rate (PER) as a more suitable metric than WER. PER is calculated in a manner similar to WER, focusing on phoneme errors in the transcription, rather than word-level errors.

For PER calculation, a high-quality phoneme recognition model is used, such as the one available at https://huggingface.co/facebook/wav2vec2-xlsr-53-espeak-cv-ft. The evaluation process is as follows:

• The test set is processed by the candidate AL model to produce the converted speech samples.

• These converted speech samples are fed into the phoneme recognition system to obtain the predicted phonetic transcriptions.

• PER is calculated using predicted and reference phonetic transcriptions.

This method addresses the phonetic precision of the AL model to a certain extent.

Naturalness Prediction

To assess the naturalness of generated speech, one common method involves conducting subjective listening tests. In these tests, listeners are asked to rate the speech samples on a 5-point scale, where 1 denotes very robotic speech and 5 denotes highly natural speech. 

The average of these ratings, known as the Mean Opinion Score (MOS), serves as the naturalness score for the given sample.

In addition to subjective evaluations, obtaining an objective measure of speech naturalness is also feasible. It is a distinct research direction—predicting the naturalness of generated speech using AI. Models in this domain are developed using large datasets comprised of subjective listening assessments of the naturalness of generated speech (obtained from various speech-generating systems like text-to-speech, voice conversion, etc). 

These models are designed to predict the MOS score for a speech sample based on its characteristics. Developing such models is a great challenge and remains an active area of research. Therefore, one should be careful when using these models to predict naturalness. Notable examples include the self-supervised learned MOS predictor and NISQA, which represent significant advances in this field.

In addition to objective metrics mentioned above, we conduct subjective listening tests and calculate objective scores using MOS predictors. We also manually examine the quality of these objective assessments. This approach enables a thorough analysis of the naturalness of our AL models, ensuring a well-rounded evaluation of their performance.

Closing

AI Accent Localization technology is a disruptive innovation, primed to bridge language barriers and elevate customer service while expanding talent pools, reducing costs, and revolutionizing CX.

Subscribe now