Advanced Chatbot Analytics and Sentiment Analysis
An introductory, hands-on tutorial on chatbot analytics and sentiment analysis.
Introduction
The last weeks brought good news to chatbot developers: Google has opened its Chatbase analytics service to the public and AWS has presented Amazon Comprehend, a new service to run topic and sentiment analysis on texts, at re:Invent 2017 in Las Vegas. Both tools are great complements for advanced conversational applications and can contribute significantly to the design of effective and successful chatbots and the improvement of existing solutions.
What is Sentiment Analysis and how does it apply to chatbots?
In a nutshell, Sentiment Analysis is an automated process that programmatically extracts topics from texts and the feeling of the writers towards such topics. It is a well-known and widely used practice in marketing and politics, to prepare and adjust communication strategies.
Chatbots are meant to automate and streamline communication with users, providing immediate responses and improving the customer experience through a friendly, conversational interface. One very common use of chatbots is first-level Customer Support. The purpose of Customer Support, though, goes far beyond the actual help provided to users: it is a valuable source of information about products and services, how do user rate them, what users like and what frustrates them.
Being chatbots “conversational” and with users still not always aware of being actually chatting with a machine, they will likely expand and share their feelings in their messages. Like well-organized Customer Support teams do, performing Sentiment Analysis on incoming messages a chatbot can collect feedback from users, normalize and aggregate data, and submit it to Product Management and Marketing for review.
Amazon Comprehend, sometimes in conjunction with Amazon Translate, it’s the perfect tool for the job. With an abundant free tier to develop and test and a favorable pricing structure (it’s $6 to parse 10,000 messages up to 600 characters each, at the time of writing) Amazon Comprehend is arguably going to be the most accessible tool for the job. In the upcoming part #2 of this article, I will show how to use Amazon Comprehend in your projects.
What are analytics and how do analytics apply to chatbots?
Modern analytics are way more than a mere count of people who accessed a service and a regional breakdown of the data. They measure how much time users spent on each interface, what call to actions did they respond to, how long their attention span was and there are even heat-map tools that can track what areas of an interface user have been most focused on and interacted with.
In a nutshell, analytics measure the quality of the user experience.
A detail as small as a label on a button can determine the effectiveness of an interface and its success over the competition. Finding and removing the friction points is the key to an high “conversion rate”, which is “the proportion of visitors [..] who take action to go beyond a casual content view or [..] visit”.
In chatbots, it’s important to observe the conversation flows, count the number of “intents” and “entities” successfully decoded over the total number of incoming messages and the number of messages that a user had to type in order for his request to be fulfilled.
The most important action to take in order to improve the effectiveness of a chatbot is to set and manage expectations.
With the upcoming self-driving cars revolution and the democratization of AI, now available in a wide range of personal devices and home appliances, it took very few years to users to become more and more accustomed to technologies that once existed only in sci-fi movies and books.
This often creates expectations in them that go beyond the current state of technology. It is an highly sensitive topic for chatbots, as many users expect to be able to talk to them like they were actual humans (and some less-tech-savvy users can’t always tell they aren’t) and can get easily annoyed if they don’t live up to such expectations.
In spite of the extensive knowledge on user and customer behaviour that we can rely on today, humans can still be unpredictable at times. Manually reviewing all conversations would be an unworkable task and in contrast with the principles of automation and conversational interfaces. Creating the perfect experience requires thorough testing, especially A/B testing, and constant analysis, validation and tuning.
Chatbase can perform very complex analyses of all the conversations between users and bots, measuring the effectiveness of each message, the expectations and suggesting optimizations in a convenient, clean graphical interface. Best of all, it’s even free.
Chatbase crash-course
Integrating Chatbase in your projects
Chatbase exposes a simple, common REST API that can be easily integrated in any application. Plus, it offers client libraries for Node.JS, Python and .NET at the time of writing.
Crunching the numbers is as easy as forwarding all the messages sent and received by the bot to Chatbase and marking them as “handled” if the bot was able to successfully process and fulfil the request. Please note that at this time the stats are not updated in real-time and it takes a few hours for the report to be refreshed. So just trust the API responses during development to test your integrations.
Note on Dialogflow: Dialogflow (once known as api.ai) users get built-in basic stats on Chatbase. This is convenient and can save you the effort of the integration, plus some network traffic and computing time in large-scale environments (especially in cloud environments where resource usage is metered and billed). In order to get more detailed stats or integrate the service in more complex bot implementations, though, a custom integration is required at the time of writing.
Technical considerations
Forwarding messages to Chatbase involves issuing HTTPS requests to its API endpoint and this will have an impact on network traffic and computing time. Such impact is hardly noticeable in a low-traffic scenario, but as a best practice all applications should be built for scalability. Also, a bad implementation could introduce latency in responses, degrading the performances for the users.
Chatbase supports sending of multiple messages per request, so collecting messages in a queue using a fast database (such as Redis) and forwarding them in batches using a background worker could be the best approach for this task.
When a background job is not desirable or applicable, it’s still worth leveraging the multi-message send API to wrap together the user message and the bot response in a single call and possibly parallelize it with the response being sent to the user.
Sending data to Chatbase with Node.JS
The following code has been taken and adapted from my serverless-bot-skeleton-with-analytics project. It's based on the official Chatbase Node.JS Client. The code below is commented on each line and should be easy to follow. It is also available as a GitHub Gist for better readability and code highlighting.
const chatbase = require('@google/chatbase');var intent = "ASK-CURRENT-TIME"; // This should be actually decoded from the user message!// Create a Message Set
// See: https://github.com/google/chatbase-node
var messageSet = chatbase.newMessageSet()
.setApiKey("APIKEY") // Chatbase API key
.setPlatform("facebook-or-whatever"); // Chat platform name
// Track the message from the user
const userMessage = messageSet.newMessage() // Create a new instance of Message
.setAsTypeUser() // Mark it as a message coming from the human
.setUserId("facebook-user-XYZ") // User ID on the chat platform, or custom ID
.setTimestamp(Date.now().toString()) // Mandatory
.setIntent(intent) // The intent decoded from the user message, if applicable
.setMessage("Do you know the time, please?"); // User message
// Was the intent successfully decoded?
if (intent == "UNKNOWN") {
userMessage.setAsNotHandled(); // Tell Chatbase to mark this user request as "not handled"
} else {
userMessage.setAsHandled(); // Mark this request as successfully handled ;)
}
// Track the response message from the bot
const botMessage = messageSet.newMessage() // See above
.setAsTypeAgent() // This message is the bot response
.setUserId("facebook-user-XYZ") // Same as above
.setTimestamp(Date.now().toString()) // Mandatory
.setMessage("It's 12 o'clock!"); // Bot response message
// Send all messages to Chatbase
return messageSet.sendMessageSet()
.then(response => {
var createResponse = response.getCreateResponse();
return createResponse.all_succeeded; // "true" if all messages were correctly formatted and have been successfully forwarded
})
.catch(error => {
console.error(error);
return false;
});
For Node.JS projects, you can grab & use the ChatbaseClient class from my project. Just replace the Intents module and intents variables with your own.
Sending data to Chatbase with Python
The following code is taken from my fork of the Python Client for Chatbase. Check my pull-requestif you are interested in the details of my fork.
Important: this code will only work with my fork of the Chatbase Python library (see above). The official library does not reflect the full API at the time of writing and some functionalities are missing.
This code is also available as a GitHub Gist for better readability and code highlighting.
from chatbase import Message, MessageSet, MessageTypes, InvalidMessageTypeErrorapi_key = 'APIKEY' # Chatbase API key
platform = 'facebook-or-whatever' # Chat platform name
message_user = 'Do you know the time, please?' # User message
message_bot = 'It's 12 o'clock!' # Bot response message
intent = 'ASK-CURRENT-TIME' # This should be actually decoded from the user message!
version = '1' # Bot version, useful if you want to mark them for A/B testing or compare results across versions
user_id = 'facebook-user-XYZ' # User ID on the chat platform, or custom ID
time_stamp = int(round(time.time() * 1e3)) # Mandatory# Create an instance of MessageSet to collect all the messages
message_set = MessageSet(api_key=api_key, platform=platform,
version=version, user_id=user_id)# Create an instance of Message for the user message and set values in the constructor
msg1 = Message(api_key=api_key, platform=platform, message=message,
intent=intent, version=version, user_id=user_id,
type=MessageTypes.USER, not_handled=True,
time_stamp=time_stamp)
# Set the message as "handled" because the NLP was able to successfully decode the intent
msg1.set_as_feedback()# Create an instance of Message for the bot response message and set values in the constructor
msg2 = Message(api_key=api_key, platform=platform, message=message,
version=version, user_id=user_id,
type=MessageTypes.AGENT)
message_set = MessageSet(api_key=api_key, platform=platform,
version=version, user_id=user_id)# Push messages into the collection (MessageSet)
message_set.append_message(msg1)
message_set.append_message(msg2)# Send the messages
response = message_set.send()
# response.status_code will be 200 if sending worked
For Python implementations, you can also consider using the HTTP API directly with the aiohttp library. It works in asynchronous mode and it is super-optimized.
Well done!
Part #1 of the tutorial is complete and now you can start practicing with your new Chatbase analytics tool. Feel free to share and discuss the results with me! In part #2 I will show how to move further implementing Sentiment Analysis with Amazon Comprehend on your chatbot.
