I Was an Undercover-Bot For 2 Months. Here is What I Learned.
Bots: hype or glory?
Bots, and especially chatbots, are a subject close to my heart as a User Experience geek. I spent years building interfaces, and now I’m told by the tech-media that apps are history, and the future is textual — talking to AIs.
That may be the case, no-one wants to be that guy who said “Nobody will buy an iPhone”, or “The world needs just 4 computers” — but, at the same time, I haven’t yet met a single bot that actually does ANYTHING better than a great UX. Scratch that, I haven’t yet met a single bot that actually does ANYTHING beyond a narrow scripted interaction.
They all seem OK inside a script (in English), but become super-annoying very fast, when moving out of their pre-designed limits. Even if a bot understands 2,000 variations of “book me a meeting” — it’s still not AI, sorry guys.
So I Decided to Deploy a Chatbot, Without Building One, Just to See for Myself.
Before jumping headfirst into developing a bot platform, I thought I’d do a small experiment with the platform I was working on.
The app is called WorkGroup (http://www.workgroup.im). It’s a simple teamwork platform used by over 10,000 groups worldwide to communicate and manage action items in real time. You can sign-in here to get some context about this post.
Now, in the communication/collaboration space, everyone’s launching bots, but we were designing a MUCH simpler platform than anything in the market. I was aiming for normal people (non-geeks), to be able to open workgroups faster to than they write yet another email, so I had to know if bots add simplicity, or bots add complexity.
So Here is What We Did:
Every new user to the WorkGroup platform (which was in beta at the time) had one group pre-opened with our “bot”. The bot’s official role was to assist users in using the app. But it had another important task — to collect real user feedback.
In the tradition of MVPs, the bot was actually something extremely basic, which alerted me every time someone started a conversation with it. I would log-in to the bot, and continue the conversation myself, as the bot. We were completely open with users, that this is a semi-automatic bot, but most people didn’t care enough to ask about it, and were happy to chat away.
The bot (with me or my teammates in it undercover) ran for almost 2 months.
And Here Is What We Learned:
First, the task was a success from our marketing point of view. Some people love talking to bots. We learned a lot in terms of what they think about the Workgroup app, various improvements that could be made to make it simpler (most are already implemented), and how well our “product-story” was being understood.
But, we also learned how incredibly hard it would be to build a “real” bot (not semi-automatic with human assistance like we did). Remember, our target users are not just tech-savvy users, we really want normal people to feel at home, and only ever meet the most simple and useful interfaces.
Here are some of the things we learned we would have to overcome, if we wanted to deploy a real chatbot:
Incredible Variation
We never got the exact same sentence twice.
(These people wanted help in deleting a chat, but just going for the keyword “delete” was not enough, because other people used the word “delete” to ask about deleting just a word, or an entire account).
Language
More than half of our users do not speak English. So even a bot that clearly states it speaks only English, and only with English speaking people, it will still have language issues: because people do not speak “proper” English. They are sometimes VERY hard to understand.
(This person was actually asking if our voice-calls feature already launched — it was.)
Length
People talk in length. Sometimes writing long paragraphs before expecting an answer. Sometimes putting together 2–5 different points in the same requests
(I removed the actual content of the comment here, it’s less important — I show it just to illustrate an example of a user who could become a top advocate of the app, with a ton of ideas and willingness to help, but way too complicated for a bot to figure out).
Emotions
People share feelings and emotions, positive and negative, they also use humor a lot when talking to a bot. A bot with a scripted sense of humor is one thing, but a bot who understands a sense of humor is a completely different ball game.
(This person was expressing emotions, which would be extremely hard for a bot to interpret and react to accordingly)
Non-Verbal Communication
People send pictures, files and use emoji. It’s true that you can block those, but again, that’s how people communicate
(An example of a user using Emoji, we also got files, pictures, funny gif images, and a ton of links).
In all, there is no bot in the market today (please refer me to one if I’m wrong here), that can manage these types of conversations convincingly.
What Can Be Done?
When talking to bot experts, I realized quickly that most of the real-world solutions to the challenges I mentioned, required limiting the scope of the bot to much more specific tasks, and limiting the interaction by ways of asking users to choose between options, or asking them to confirm that the bot understood the request. Sounds too much like those annoying voice machines that answer me when I call my bank.
Is There a Middle Ground Here?
Absolutely. Both sides will learn. Bots designers will learn how to keep a conversation fenced.
Humans will learn how to speak to bots (hint, not the same as speaking to other humans). My guess is it will take one extremely successful mainstream killer app to teach a complete demographic how it’s done, and from there it’s just a matter of time.
Are We There yet as an Industry?
Not even close.
The bots we have today, would maybe function OK in a limited environment, and with super-patient users. But unleash such an app on the open market — and normal users will just find it frustrating and complicated.
Some platform have seen some success with limited bots, but I suspect it has to do more with an audience of tech early-adopters who are willing (and happy) to memorize command-line style interactions.
Conclusion
For now, we are keeping WorkGroup super-simple, clean, and bot-free — that’s how our users like it.
The welcome group is still there for new users — you can see it if you click here: web.workgroup.im , but now instead of the bot, you will meet Tom, our support guy. He may be a bot too. Let’s see if you can tell the difference…
Follow me on Twitter: @amibendavid
