A Product Manager’s Guide to Building Your First Bot

Published in

Chatbots Magazine

13 min readOct 18, 2017

This is one of two posts sharing our learnings from launching a bot. The first was a guide do launching your first bot. This second highlights how we found product market fit with our bot.

So you think your company needs to build a bot, and you’ve been tasked with making it happen. This guide reflects the key decisions and questions I had to answer as the Product Manager to launching our bot in six weeks.

TL;DR

50 questions you need to answer when building your bot, which roll up to these 5 top-level questions:

What’s the problem and how will you solve it?
What will the bot do?
How will you build the bot?
How will you know if the bot is successful?
How will you launch the bot?

Part 1: Set Scope and Strategy

Your first priority is to define and articulate the scope and strategy of this product. Frankly, defining the problem and your solution should happen for all your projects, not just for bots. The questions you want to answer are: What is the problem you are trying to solve? How will you solve the problem? How will you measure success?

What is the problem?

“Unsolved problems are where you’ll find opportunity” Peter Thiel

Start with defining your problem. When you’re launching a new product, unknowns are everywhere. Defining a crisp articulation of the problem enables you to focus and align the team around the same problem.

Don’t build a bot because you think it’s the the new hot thing (because it might cool down in the next minute). This can be a risk for an established business as they’re building based on trends, or “me too” play, without spending enough time on articulate a clear problem to solve. Arte Merritt, CEO of Dashbots, an analytics platform for chatbots, notes that in general brand bots don’t as well as independent bots. While scoping it is critical that you spend the time spelling out your known knowns, known unknowns, and unknown unknowns.

Scope the Problem
To determine the problem to solve, answer these questions:

Who is it for?
Is this a pain or a desire, and where does it fit on the scale?
How many people feel this pain?
How many people have agency to act on on it?

Take for example, getting stuck in hour-long traffic is really painful for many. But many people don’t have theagency to act — one can mitigate it through dynamic rerouting, like Waze, or have the authority/funds like Google/Tesla to spend a lot of money on autonomous driving. Understanding this helps to map out how difficult it may be to solve.

How you will solve it?

“There are only two ways to establish competitive advantage: do things better than others or do them differently.”
Karl Albrecht

With a problem in hand, the next step is to determine how you are going to solve it. What will separate your product from the competition is if you can find a unique value proposition that separates you from the pack. Three questions can help you get there:

Why you?
Why now?
What do you believe?

What is your advantage?
Start looking internally at you, your team, and your company to take log of what makes you unique. Building a bot is hard. Not only because of the development time, but because you need to find a way for the user to discover and install your bot, much less retain the user (40% of chatbot users only engage in one conversation).

Identify why you are uniquely able to solve the problem. Your strengths are your launchpad. Our bot at Clarizen is an extension of the core product’s experience. Further, we realized that a unique advantage is that we’ve had over a million users finish their projects with Clarizen, and with that knowledge, we had a competitive advantage in understanding best practices of project management at enterprise-scale.

Why now?
Is there urgency behind why you need to build this now? Products success can be attributed to perfect timing. Take, for example, WhatsApp or WeChat, these chat apps grew due to a window of opportunity; they provided a free alternative to having to pay SMS fees. For us, a driver is the commoditization of NLP technology. Our bot’s AI engine is run by API.ai (https://api.ai/) providing the ability for project managers to type simple queries to get answers on complex projects..

What do you believe?
The best weapon for the product manager is having a unique point of view. The bedrock of developing a unique POV comes from deriving good, edgy hypothesis. A simple framework is this madlib:

Spend 20 minutes by yourself writing as many of these sentences down as possible. Yes. 20 minutes. The first few are pretty soft. There comes a point in time, where you’ll hit a wall and run out of ideas. Keep pumping them out.

When you finish, review all the hypothesis and prioritize. A compelling hypothesis is one that if true, would have a material impact on how you design the product.

How do you know if you have solved it?

“Count what is countable, measure what is measurable, and what is not measurable, make measurable”
Galileo Galilei

Congratulations, you’re almost finished with the first stage. Now that you have identified a problem for a target market, and how you will uniquely solve it, you want to set some metrics in place to know if you’ve reached your goals. Many people have written about finding the right metrics (Some of my favorites: Alistair Croll, Josh Elman, and Kevin McNally/Nick Odlum)

For many of us, you’re product is successful if you can find people who want to and continue to use it. Broadly speaking you can track value in either usage (engagement metrics like DAU/MAU or time spent) or happiness (NPS).

Part 2: Define your Bot’s Scenarios

“You make different colors by combining those colors that already exist”
Herbie Hancock

In Part 1, you’ve laid the groundwork, parameters, and strategic direction of the bot. In this section, we’ll get into the more unique aspects of development the bot.

What are my bot’s key scenarios?

As a business utility app, the purpose of the bot is to as quickly as possible deliver on the intent of the user. In our case, it’s to interact with project data. Prioritize scenarios based on what your target user will find most valuable.

We started with user stories, but it ended up being a bit clunky. Instead, Arie our Engineering Lead suggested, “Why don’t you just list what the user is going to ask and what they expect in return?”. Focusing on the bookends of the conversations helped us understand the trigger and the goal. This technique was also great for brainstorming with our early customers. “What would you say to the bot? And what would the bot respond?”

How can I prioritize my scenarios?
With all the potential scenarios that you can use, I apply a framework of which I’ve modified from Kano to something I call the 3Ms: Must, Maybes, and Marvels (I’m a former consultant and I’m predisposed to find alliteration).

Must: You need to have these features. These are fundamental features that without the user will think that the product is lacking. Building a car, you probably must have a door, steering wheel, or wheels (however, there are always exceptions)
Maybe: These are features that might work for you. You may have seen it in other bots. As a PM you need to critically assess the feature relative to your problem and strategy. To assess whether or not it makes sense for your bot, compare it against your hypothesis and problems you hope to solve. This will help you prioritize. You’ll consider different options whether you’re building for the car enthusiast or the dad of three.
Marvel: These are the scenarios that people get excited about when they see. Some call it customer delight or aha moment. If you were to demo this to someone, you’d want to end on this to serve as the high note. This could be ludicrous speed or a dryvac built into your minivan (did I mention I own a Honda Odyssey?)

What are some Must scenarios?
For the must category there are a few foundational elements a day bot should support. Many of them are outlined in Amir’s book (an amazing resource), Designing for Bots. There will be a series of scenarios that you’ll need to build for that are specific to your bot. Here’s a list of features that most bots today support — many of these features are pretty common sense and seek to nudge the user back towards the happy path.

Help: Tell the user the most popular commands. We also use this is part of the FTUE flow.
Error script: While you try to avoid dead ends with your bot, you should always have a fallback error message. It’s usually a simple, “Sorry, I didn’t get that”
Feedback: When user types feedback, they can have a direct line to the product team.
Bug: User can also submit bugs (this was less for users, but something we built for our team internally while we were dog fooding it)

How do I discover some Marvel scenarios?
Chat interfaces enable unique interactions. When the user asks the bot for information, at a base level, the bot needs to provide the answer. During the answer, the bot can also provide additional information or insight related to the user’s intent. Shailesh Shilwant, from Kritii Designs, says, “Here’s what you asked for, but also…”

How does it flow?

The next step is to convert the scenarios into flows.

With chat apps, users can easily go off track. Unlike web or mobile where navigation is structured, with bots users can say anything anytime. We’ve gotten random messages like, “the click of death… now on your TV!”

What is the Happy Path?
There some great advice on how to get started. In general, start with the happy path for each scenario. The happy path is the perfect interaction and script between human and bot. I love using Walkie Botto mock up these conversations. (Though I hear Botsociety is another great tool).

How can things go wrong?
Practically, your users rarely follow the happy path. Invest the time to help nudge the user back to the happy path. Google provides a really great, disciplined way of layering complexity on your conversation flows. Generally speaking consider the following elements:

Start with the happy path. This is perfect alignment between customer actions and your designed intent.
Expand to the long path. More often than not, there’s a huge gap between what you’ve designed versus what the user sees. Assume that it takes the user twice as long to get to the payoff.
Provide hints to guide the user back to its path. There are some easy ways to help the user stay within guardrails. Visually, you can use buttons or quick replies (FB Messenger and Slack supports this) to nudge them in the right direction. Additionally, you can progressively show information to the user. For example, with the Clarizen Bot, if the user asks for a status on projects, since the bot doesn’t know which one, we segment all projects by status (Red / Yellow / Green) making it easier for them to identify which project needs their attention.
Lastly, make sure that the user can leave the current conversation flow. Whether it’s typing “quit”, timing out, or even erroring out. The bot should always allow you to exit gracefully without making the user feel guilty.

Part 3: Bot building blocks

As a Product Manager, you set a framework and outline the questions to ask. Here are some tactical key questions we had to answer when building our Bot.

What should be the first channel we build on?
As a business integration bot, the strategy that we plan to pursue is a horizontal one. If our users are on Slack, we needed to build a Slack bot. If the users are on MS Teams, we will be there. Factors we considered: market, developer friendliness, growth potential, user’s attitude in trying new things.

What should be the Bot’s tech stack?
While this responsibility can be led by Engineering, the PM needs to approach this from a strategy and business point of view. For example, the decision to bake NLU into the bot should be predicated on whether there’s a good value driver for it.

How does the bot interact with the user?
If the tech stack is the back-end, the front-end experience for the user will manifest in the bot’s personality and tone. Should the bot be silly, irrelevant, or professional? Things aren’t so clear. Prior to Slack we had always assumed the enterprise-focused apps need to be “professional”.

Part 4: Monitor Health of the Bot

By now, you’ve probably started building the bot and are starting to consider how to launch the bot and how to manage it once it’s live. It’ll be important to start setting up metrics.

Beyond the typical metrics like engagement or retention, there are a few bot-specific metrics you should consider:

Confusion Triggers: The entire funnel is wrought with ways for things to go wrong. Given this range of possible user inputs, bots often misinterpret or simply can’t determine what the user wants. Closely track when the bot responds with an “Sorry, I don’t understand”.
Conversation Steps: Knowing the average conversation steps enable you to start segmenting good or bad experiences between the user and the bot. A conversation step is a single back-and-forth exchange between user and bot. Your average steps will be different depending on the type of the bot. İlker Köksal, Co-Founder of Botanalytics notes, “Utility-driven chatbots have lower average conversation steps versus entertainment-driven chatbots. Regardless of the chatbot type, conversations that either significantly exceed or fall short of the average conversation step usually indicate a bad user experience. Either a user gave up too quickly or a bot took too long to complete a user’s goal”
Conversations Per User: Associated with the number of average steps is the average number of conversations per user. This indicates engagement with the bot. Köksal notes that the average conversations per month range from 1.42 to 4.79.
Active vs. Engaged Users: If your bot supports both push and pull interactions, then you will want to compare the differences in how users interact with your bot. Active users will consume messages sent by the bot, whereas engaged users will respond. This is important because you can learn from users who are engaged. These are your happy users that are leaning into the experience. You will want to accommodate these types of conversations.
Response Time: Know how long it takes for your bot to respond. Know the latency period between command and response. This impacts the overall user experience. You’re looking for median time as well as outliers here.

Just like web and mobile analytics you have a bunch of options. You can go with 3rd party providers (Dashbot and Botanalytics) or you can roll your own.

Part 5: Marketing

As you get ready to launch the bot, there area few decisions you’ll need to make. Beyond the standard marketing components such as product positioning, messaging, collateral, and recruiting launch partners, there are a few bot-specific factors to consider:

How do you plan to raise awareness?
There are multiple ways to raise awareness from the bot. You’ve got standard marketing and social channels like Email, blog post, or twitter. Also consider product and bot specific channels like: Product Hunt, Bot Directories (like Messenger’s, or general directories like botlist), and developer channels (like dev4slack).

What should the bot’s name be?
The strategy and the supported use cases will help you with the name. Overall, there are 4 categories of names:

Branded Name: The bot inherits the company brand as the bot serves as an extension of the core product. This creates a consistent brand between the company and bot. However, it’s not really the most compelling from a marketing perspective. Further, if you plan to have more than one bot, it creates complications — what are you going to call the second Jira bot? Examples include: Jira Bot, Google Drive Bot.
Shorted Name: The bot is a play on the company’s existing brand. Much like United launching Ted as its low cost airline. Consider this strategy when you are extending the existing branding but it’s a wholly new product.
Functional Name: The name perfectly describes what the bot does. A friend from South Africa told me that Americans tend to call things by their functional purpose. “It’s not a bin, it’s a trash can; not a rubber, but an eraser.” Consider this is you are considering a horizontal play and you have little brand recognition. Example: Pipedrive’s Dealbot, Kuoll’s BugBot, or CareerLark’s Micro-feedback Bot.
Person Name: Some companies will give their bot a name. Consider this strategy if you’re a B2C bot that needs personality (I’m looking at you Poncho), or your a B2B company looking to monetize the product. This also has a bit more marketing zing (for now). However, this means that you’ll need to invest marketing dollars to build out the new brand.

How we found Product Market Fit with our Chatbot

Lessons learned from building an enterprise bot

medium.com

Coda

Building a bot these days is relatively easy. There are so many off the shelf pieces that have been open-sourced or free services you can readily use to get started. For example, Diagram Flow for our NLU, and Dashbot for our baseline analytics to launch an MVP product in 6 weeks.

What will make your bot unique (especially when you are just starting out), isn’t the technology. It’s going to be how the bot interacts with the user and what it has to say. To create compelling bot interactions, start with a clear strategy, invest time in identifying and prioritizing key scenarios, map out ways to control and optimize the conversation flow, and discover how to delight the customer through tiny moments.

The second key differentiator will be your ability to find a unique data set and turn it into something that is useful for the customer. At Clarizen, our customers are running thousands of projects through our platform. This means that not only can we provide a personalized experience on helping the user with their individual projects, but we can use the entire user-bases data to optimize how to get their projects done! For us the Bot is the medium by which that message can be delivered.