Continuous Automated Testing for Your Chatbot With Open Source Tool “TestMyBot”

Florian Treml
Chatbots Magazine
Published in
4 min readMar 8, 2017

--

“TestMyBot” is a new test automation framework for your chatbot project. It is unopinionated and completely agnostic about any involved development tools. Best of all, it’s free and open source.

Update: “TestMyBot” has been renamed to “Botium”

2019/01/20 Chatbots are driving the industry. With Botium we are driving chatbots. After months of hard work we are finally there: we are proud to announce the general availability of the newest Botium Stack member, the Botium Box. Check out some game changing features for testing and training of chatbots with our Community Edition!

Typical code structure for Chatbot projects

Let’s face it, a software project without continuous integration and without automated testing is archaic. (Can you tell I’m a huge fan of automated testing?) Especially when I’m jumping between projects, I need a safety net in my development process that only automated tests can provide.

With my last chatbot projects Itipu and Oktocheck, I was in desperate need for a test automation framework helping me to test conversation flow.

There are plenty of excellent testing frameworks available for Node.js (like Jasmine or Mocha), and I use them whenever it’s time for unit testing my code. But it’s really, really hard to test larger portions of code with them.

  • Working with Stubs, Spies, Doubles (see Sinon) is fine for testing single methods, but not feasible if going further
  • Every single piece of code has to be written so that it can be tested (for example using dependency injection)
  • As rule of thumb, a unit test should only test exactly one function, which doesn’t fit well with a conversational flow

Thus, the idea for a test automation framework was born, and I named it “TestMyBot.”

You should know that I love Appium — it enables every Smartphone App to be integrated into an automated testing workflow.

My vision for TestMyBot is “to be the Appium for chatbots”.

More detailed, my goals for TestMyBot are:

  • Agnostic of any development libraries, Botkits, etc
  • Agnostic of any test support libraries (Jasmine, Mocha, Chai, Sinon, you name it)
  • Agnostic of build tools and continuous integration tools and platforms
  • Unopinionated about coding style
  • No code modifications required

Basically, you take your chatbot as is, define test cases for your conversational flows, and TestMyBot runs them on your chatbot.

Show Me Some Code

Enough theory — what does it look like? As a demonstration, here’s a simple chatbot:

  • When you say “Hello!” it replies with “World!”
  • When you ask “How are you?” it tells you “Fine, and how are you ?” In the next step, the chatbot will repeat your answer to this question (“Nice to hear you are <whatever you answered>”).

The conversational flow definition would look like in this Jasmine test spec.

describe('Hello World Bot', function() {  beforeEach(function() {
this.bot = require('testmybot').setup();
});
it('says hello', function() {
expect(this.bot.hears('Hello!').says().text())).toMatch(/world/);
});
it('asks how i am', function() {
expect(this.bot.hears('how are you').says().text()).toMatch(/and how are you/);
expect(this.bot.hears('I am fine').says().text()).toMatch(/nice to hear you are fine/);
});

});

You can see that the “bot” object is used for sending text (or postbacks, or media) to your chatbot, and to receive whatever your chatbot replies (text, rich messages). It can be operated with any test framework (or even without any test framework).

What Else? API Mocks!

The TestMyBot framework also contains a simple API Mocker (currently HTTP/HTTPS APIs only). In this example, any request to the OpenWeatherMap API from your bot will be answered with the response you define in the setup.

exports = {
...
apimocks: [
{
name: 'weather',
urlpattern: 'https://api.openweathermap.org'
response: {
.... // whatever should be returned
}
}
]
...
};

For a more dynamic approach to API mocks, you can define a response function which will be called instead of the original API.

Nerd Talk

If not interested in technical details, please skip this section.

TestMyBot heavily relies on Docker to provide it’s outstanding capabilities in test automation. Your chatbot is fully transfered into a local docker container, the API mocks are possible by manipulating the DNS of the docker image. TestMyBot opens a channel to your Chatbot webhook and answers your Chatbots requests to the messenger API, providing everything to your test specs.

Current State and Outlook

For now, there are some restrictions with TestMyBot.

  • While TestMyBot can be used with any Chatbot running in a Docker container, continuous testing is currently only working with Node.js projects.
  • Currently, only Facebook chatbots using the Facebook Messenger Platform directly are supported.
  • Development is ongoing, Slack and Python support are on the way. Please join my Github project for progress.

--

--