How we got a 98% success rate for our bot for Zalando — A UX case study
Zalando, one of the largest e-commerce companies in Europe, was holding their inaugural Vizions event to announce their new platform. They approached us at avocado to create a new chatbot for the event to improve communication with their attendees while showing the world that they are leaders in technology.
We took a UX-driven approach to research, define and create a successful bot on Slack.
The Problem
Last minute changes or confusing schedules, being forced to download a new app, getting lost at a huge event space, not being able to network, can’t find the wifi password, do any of these sound familiar?
We were tasked with creating an engaging new chatbot to provide information to attendees in a way that was more modern and efficient than an app or website.
Goals and Objectives
- Increase customer satisfaction.
- Lower customer service costs.
- Improve communication between attendees and with the conference.
Final Results
The Process
Currently, there is no specific process for bot design so this was immediately an exciting project for us! The internet is filled with articles on bot personality but there is so much more to the design process than just that. We took a UX-driven approach that was similar to designing an app or website.
Bot personality is only a minor part of the overall bot UX.
Research
We collected the questions that were asked in another Slack channel for another conference and also talked with event organizers to see what the most recurring questions from attendees were. Consistently the most recurring questions could be divided into 3 main categories:
1. Schedule
2. Map
3. Tickets.
When we did a further survey, we found the pain points were mainly in the area of confusing schedules and maps, but also the frustration from being forced to download a clunky new app just for the event.
Pain Points
- Unnecessary downloading of apps: Being forced to download a “crappy” app only for the conference.
- Schedule changes: not having easy access to the schedule, or confusing schedules.
- Navigational issues: Unable to access the floor-plan and finding it hard to find certain areas.
- Lack of networking opportunities: Not having a way to network as much as they’d like to and feeling like they’ve missed good connections.
Sketching
We started to sketch out some ideas of how the flow might work.
After discussing and iterating on these ideas with our team and stakeholders we then moved to mapping our conversation flow journeys.
Task Flow
From here we laid out the formatting of the dialogues in Sketch for examples of what the conversation may look like. It is crucial to ensure you have a limited amount of buttons and text so that it visually does not cause cognitive overload. This was just the MVP that we would continue to iterate and add to of course.
As Slack didn’t have a UI kit available, we designed one for us to use and have made available for free for others to use here!
User Journey Flow
We laid out the flows of what it could be for the user:
From our research, it was evident we had to give the user quick actions to access the schedule and floorplan to create an initial linear flow and guide the user starting with the Welcome message.
Our final Welcome message we included in the bot (after lots of iteration 😉):
Content Spreadsheet
We created a spreadsheet to keep all the bot responses to make it easier to work with Zalando’s marketing content team. This helped to create a brand voice, to ensure all information was up to date, ensure consistency as well as keep the same formatting (lettercase, caps, etc.).
From this sheet, we had to manually enter everything into api.ai. After the project we decided we would switch from api.ai to Rasa; Rasa has a better .json format so we’d be able to extract the data from the sheet and import it smoothly.
NLP 💃🏼
With bots, the more you add to the bot the more conflict there will be. The more features you add to the bot the higher the chance of the bot breaking as well. Hence, we had to ensure we hit the right mark.
Train phrases over keywords.
As an example, if the user asks “where is the main stage?” we can’t add “main stage” as a keyword because this would create conflicts between “where is the main stage” and “what is the main stage schedule”. If the user just said “main stage” we had to have the bot ask them “schedule or map?”.
We needed to nail down the top 10 questions for the MVP based on our research and stakeholder requirements to ensure we had all possible questions the attendees had but without making too many conflicts or making it too complex. These questions would continue to be removed/edited/added as we build and tested the bot.
Bots need to be constantly trained to understand humans. We had to get users unfamiliar with the bot to ask questions in different ways like non-native English speakers, people from different countries and having people write typo’s.
We were lucky enough to have 100s of people at Zalando play with the bot so we could train it in the backend and test. If we didn’t have this luxury, we’d crowdsource ways to say things in a spreadsheet from different people using Upwork or Amazon Turk or have other users play with the bot.
Ie. just a few ways to say “Where is the washroom?”:
In addition, we trained the bot on conversation that should be included in all bots such as swearing, small talk, emoji’s, asking for a human, emergencies etc.
And of course, we had a fallback that included options for the user to get back on track with the bot and suggest what they can do. If the bot doesn’t understand three messages in a row, it will ask if they would like to connect to a human. We built an interface to view the conversations and notify a human to respond to the user if they wanted to speak with a human.
Quick brag: luckily during this project, none of the users wanted to talk to a human as the bot was highly successful with a low fallback rate. 🚀
User Testing
We conducted user testing the same way you would for apps or websites. We used Zalando’s usability lab and had a user researcher ask our questions we assigned them as we analyzed and watched the video in another room.
I won’t describe our full user testing process but if you’re interested please feel free to read another one of my articles.
As always, we discovered a lot of exciting changes minor to major for the bot. User testing was extra helpful because after we tested it, we trained the bot to understand the testee’s phrases.
A few examples of interesting findings and changes we made:
- We discovered users wanted to search by topic versus just stage. So, we added this as an option after the user chose “schedule” they could tap the button to sort by.
- People love food! Users kept asking about food which we never thought to include. So, we added an FAQ question answer about food information.
- Users scrolled back up more than expected to view previous things they have discussed with the bot. Since the maps were so large, it made the scrolling more tedious and the conversation confusing. So, when users asked where certain locations were, we explained with text then had a button to view the map image. This was particularly interesting because we would have only discovered this in in-person user testing and not beta testing by just reviewing the conversation data.
Beta testing and bot training
It is crucial to get users to use your bot prior to launch.
We sent the bot out to be tested by a group and had a survey afterwards to ask about how the experience with the bot went while monitoring the conversations on our end.
We trained the bot and added all the language, words, phrases the user said. We tracked if there were any other FAQ requests to the bot that we may not have thought about.
End Result
By the end, the bot was highly intelligent with a low fallback rate! Attendees were happy and constantly using the app throughout the event and Zalando was extremely happy with what we produced.
Here are some screenshots of just some of what the bot could do in the end:
** Update: I have now left avocado bots to focus on Conversational UI and Product design, visit my website at jayceeday.com. Want to work together as freelance or hire me full-time? Email me at hi@jayceeday.com. 😊
Big thanks to Alex, Nikkita, Lauren & for editing and keeping me motivated. 💕
Did you know you can give up to 50 claps by continuing to press the hands? Please give 👏 ‘s if you enjoyed reading it means a lot!
