How Computational Linguists Help Your Chatbot Understand Humans

Joe Lobo
Chatbots Magazine
Published in
4 min readAug 30, 2017

--

Formula 1 cars are powered by a robust engine, capable of reaching unheard of speeds for the common man. But the best cars need a skilled driver in the cockpit capable of maneuvering this dominant beast on the trickiest circuits.

If natural language processing (NLP) is the chatbot’s engine, then the computational linguist is the driver.

What Does A Linguist Do Exactly?

At Inbenta, the linguist essentially ensures the chatbot is answering correctly by maintaining the quality of the knowledge base.

But this is a very generalized summary of their role. To continue the racing car analogy, the tasks carried out during testing and qualifying are different to when the racing itself begins.

What does a chatbot without NLP and a computational linguist look like?

Setting Up The Chatbot

When a linguist is assigned to develop a chatbot, the first question, of course, is what language are we working on — or more importantly what type of language is it.

Mandarin, Russian and English each have different semantic relations that need specific tools to develop the knowledge base. Inbenta works with 25+ languages which have already been developed and are ready to be implemented in a new chatbot (minus a few tweaks outlined below).

However, requests do come in for less common languages which might need new products to handle it.

Once the requirements have been ascertained, the linguist then develops the lexical resources for that particular language: the spelling, correction rules, providing solutions to ambiguities in the language and so on.

What Is A Lexicon And How Is It Developed For Each Language?

Simply put, the lexicon is the vocabulary that we use in everyday lives. In this case, it will be the chatbot’s vocabulary, what it will recognize and how it will communicate with clients.

The two stages to creating an ideal lexicon for a company chatbot.

When each language is developed for the very first time, the linguist will have built a general and local lexicon.

The general lexicon refers to the wider vocabulary of words and phrases that we use on a daily basis within natural language. Imagine you are meeting your friend for a quick drink at a bar — unless you have a particularly unique set of interests you will be using this language.

The local lexicon is created to adapt to your industry’s specific terminologies to ensure it meets the exact requirements of the user. For example, a banking chatbot might need to understand overdrafts, mortgages and ISAs.

Throughout this process, the linguist will be in communication with the client to ascertain exactly what other unique phrases might need to be recognized. It is at this stage when they will add the final touches to the lexicon. This generally involves phrases and acronyms specific only to that particular company — think Big Mac for a McDonald’s chatbot.

In addition, the linguist and client will discuss what jargon is not required for this particular chatbot. For instance, the banking chatbot might not need to recognize the difference between simple and compound interest. Like teaching a pig to sing, it wastes your time and annoys the pig.

Is The Linguist’s Job Finished When The Chatbot Goes Live?

Not quite. At least in the early stages, the linguist will monitor the questions and answers within the chatbot’s database in order to fine-tune its ability to match the content with the right answers. Therefore, improvements can be made to further enhance the chatbot’s accuracy.

For example, it could turn out that a large number of people are asking the banking chatbot about how to extend their overdraft and are instead receiving an answer about the overdraft available for each account. After performing a gap analysis, the linguist can recommend adding an answer to facilitate these inquiries.

As racing cars are constantly tweaked and refined with new innovations, so to the linguist will continue to add to the chatbots capabilities to further improve its self-service rates. There might be a chequered flag in Formula 1 but there certainly isn’t when it comes to developing the best bot possible.

Anyone interested in becoming a computational linguist can find out more here.

Inbenta provides chatbots powered by patented natural language processing for companies around the world including Ticketmaster, Gol and Allianz.

Enjoyed reading the article or do you just want to send us some love? Please click the below to share it with other readers!

--

--

Content creator for WPML and Toolset. Explorer of tools to improve business and countries. Blogging here https://wpml.org/blog/