Our August meetup featured Midam Kim from ServiceNow talking about the challenges of voice agents and AI. Midam opened by sharing some extremely relatable stories about how frustrating it can be to try and interact with voice customer service agents…

While there’s been major improvements in the last few years, these systems still have a long way to go. As Midam said, human communication is a joint activity that requires both parties to be active participants. To truly be useful, voice agents need to be competent participants in the back-and-forth that represents a useful conversation

In other words, despite having much more intelligent “brains” behind them when compared to the old Interactive Voice Response (IVR) systems (you know, the old “press 1 for billing”) and virtual agents (e.g. “you can say things like ‘track my order’ or ‘check my balance’)… today’s systems still leave many people shouting “please let me talk to a human!!”

Midam laid out many reasons for this: from technical failures on the audio engineering side to the fact that agents have limited (at best) ability to understand emotion. Please see her slides below for the full framework she shared!

Us non-linguists may be thinking to ourselves, “Why is this voice agent so dumb? When I talk to ChatGPT it doesn’t seem so dumb??” Midam’s framework highlighted the fact that while voice is the most natural method of human communication, it comes with huge amounts of additional data and context beyond just the words. A text-based chat: takes clear turns, isn’t real-time, allows both sides to easily see the full historical chat log, etc.

If your voice system goes: speech-to-text -> LLM -> text-to-speech, then the LLM is missing lots and lots of information. Assuming this pipeline works perfectly (which it of course doesn’t), the text that comes into the LLM is missing emotion, context, emphasis, timing, and all the other meaning we imbue into spoken language that just can’t make it into a transcription. On top of that, there’s huge variation in the way different people speak, even just within one language like English!

If that sounds like a daunting problem, linguists like Midam are helping things to improve. While for better and worse it may be a long time still until a voice agent can replace a competent human agent, it’s clear that with the help of AI that voice agents are getting better and better.

We’re taking September off, so we’ll see you in October!