In the world of AI, the concept of the "black box" — a situation where the inner workings of an algorithm are not easily understandable to humans — is simply accepted. But while it’s not always clear to users why or how AI systems make their decisions, measuring their success and improving their performance shouldn’t be a mystery.
Especially because measuring the success of your automated system is crucial for maximizing its potential ROI. Without clear insights into where automation is succeeding and where it falls short, businesses risk missing out on opportunities for improvement and growth.
And that’s a real problem for today’s CX professionals. In fact, 31% of support leaders report “limited reporting capabilities” with their current tech stack.
This is why we’ve improved our analytics suite: to address these challenges and empower businesses to measure automation success effectively. We’ve developed features such as Bot Satisfaction Score (BSAT), Resolution States, and Conversation Journeys that you can leverage to gain valuable insights into the effectiveness of your automation efforts. Then you can use these insights to make targeted improvements to your flows and your overall customer experience.
So without further ado, let’s dive into each of these features and learn how they simplify measuring CX automation success.
Learn how analytics factor into Ultimate’s tailored approach to reaching +60% automation rates.
Directly compare your bot and your agents with Bot Satisfaction Score (BSAT)
Everyone in customer service knows how important customer satisfaction or CSAT is. But we’re calling it now: BSAT is the hottest new metric on the block.
So what exactly is BSAT? It’s basically a way to measure how happy your customers are with their bot experience. Platforms that offer BSAT — like Ultimate — prompt customers to rate their experience on a scale of 1-5, or an emoji scale of star-eyes to sad face, if you prefer.
This means that you can measure your bot performance the same way you would your human agents. Unlike traditional metrics, BSAT offers a standardized measure that allows for direct comparison between human and bot interactions. What’s more, it provides actionable insights by highlighting trends and areas for improvement within specific intents or conversation flows.
Take control of how you measure automation rate with Resolution States
Automation rate may seem like the ultimate metric when it comes to measuring automation success. But what if we told you that most “automation rates” don’t accurately represent the rate of automation?
Stick with us here. Many platforms are simply calculating deflection rate.
Any conversation that does not end in escalation is “deflected”. However, this can include conversations where visitors dropped off before an escalation or were unhappy with the resolution.
Within Ultimate’s platform, Resolution States offers a way to have more control over what you deem “success” to be — resulting in a more accurate and personalized automation rate. By assigning different states to the conversation flow — like informed, resolved, escalated to an agent, and more — you can accurately track the progression of customer queries and interactions.
Then, by combining conversations that end on Informed and Resolved, you can calculate your automation rate — the percentage of chats that reached an automated resolution. Not only does this offer you a more nuanced way to measure automation, it also unlocks deeper insights that allow you to drill down into conversations.
Dive into dialogue performance with Conversation Journey Explorer
So now you’re able to measure how happy your customers are with your chatbot and you’ve accurately defined what a successful automated conversation looks like. Next up: determining where your automated conversations aren’t successful and determining how to improve them.
To do this at Ultimate, we’ve developed the Conversation Journey Explorer which allows you to review dialogue performance in order to optimize conversations and improve your automation rates. You can deep-dive into specific metrics at the Intent and Reply level to understand how your replies are performing. By tracking customer journeys and identifying key drop-off points, you’re able to uncover patterns and insights to improve the overall user experience.
For instance, one of our customers noticed that customers were dropping off after the first message because the bot didn’t support the language that their customers needed. By adding support in that language, they were able to drastically reduce their not understood rate.
By regularly reviewing this aggregate view of your bot performance, Conversation Journey Explorer allows you to optimize conversations and improve automation rates overall.
How these metrics work together to improve CX and demystify AI
When used together, BSAT, Resolution States, and Conversation Journey Explorer give you unparalleled insight into your AI platform’s performance.
Here’s how they all work together:
- Resolution States is at the core: It allows you to determine which part of the conversation is informed and resolved and therefore measure the success of automation according to your own criteria.
- Conversation Journey Explorer builds on Resolution States: It clearly shows how each conversation ended (i.e. which resolution state the conversation reached).
- BSAT can be triggered by certain Resolution States: This gives you the most control over when it’s triggered and offers a direct measure of customer satisfaction with your bot.
You can use these insights to discover the bot’s strengths and weaknesses, then tweak what’s not working — and scale what is working — to increase automation rates. This improves your customer service by giving your customers better, faster, more accurate automated support.
Why measuring support automation success is so important
Being empowered to effectively measure your customer service automation platform’s success is essential for enhancing customer service, increasing ROI, and alleviating the burden on support agents. By leveraging tools like Resolution States, BSAT, and Conversation Journey Explorer, you can gain valuable insights, identify opportunities for improvement, and ultimately achieve automation rates of 60% and beyond.