Playbook for Testing Chatbots

The free guide for you to understand what your chatbot is doing wrong.

After a couple of years involved in UX chatbot design at BEEVA, after a bunch of prototypes, and after infinite hours testing other people chatbots I partnered Nieves Abalos and Carlos Muñoz and started a side project called chatbottest.com.

Chatbottest is an open source Playbook of 120 questions (and counting) that you can use for free to test your chatbot and its UX. Similar to what you get with an Heuristic Evaluation on traditional interfaces, with this guide you will be able to find out what users will expect from their interaction.

In this article I will share our findings and conclusions working in that guide. That way we’ll help you understand why testing is important, how you can create your own tests and where you should focus depending on your situation.

If you are looking for a conversational and VUI designer reach me at jesusmartin.eu

Why is chatbot testing important?

In all this time, I’ve talked with lots of people and tested hundreds of chatbots and I’ve found three important insights regarding chatbot design.

• Lack of objectivity: We tend to be really excited by our chatbots doing “something”, even if that “something” is not doing anything valuable at all. Besides some direct metrics like users, returning visitors, or successful requests, we don’t have a way to measure the quality of chatbots.
• Responsibility: It’s still a really new field with only a few designers involved and most of the (great) work has been done by developers. There are not many specific processes or teams dedicated to the duty and it is not clear who should be taking design decisions and how those decisions should be taken.
• Focus: Chatbots are really exciting and it’s easy to get lost in functionalities and developments that doesn’t mean a real improvement for our users.

What are the consequences of not understanding where your chatbot is failing? It…