Delbot: Building An NLP-Based, Voice-Driven Bot From Scratch in Python

Delbot understands your voice commands, searches news and knowledge sources, and summarizes and reads out content to you.

Published in

Chatbots Magazine

7 min readFeb 18, 2017

TL;DR: If you want to dive straight into the code, you can head over to Delbot, my GitHub repository for this project.

Check out the demo video!

Introduction

Bots remain a hot topic. Everyone is talking about them.

How about building one from scratch? The simple one we will build today will understand and answer questions like:

What is the latest news on Star Wars in the New York Times?
Who is Donald Trump?
Read me the latest on Brexit.
What are RDF triples?
Who was Joan of Arc?
Give me news about the UK government from the Guardian.

Our goal is to code a bot from the ground up and use nature language processing (NLP) while doing so.

In addition, our bot will be voice-enabled and web-based if you complete the web app section as well. The best part is we do not need to do anything fancy for speech recognition and synthesis: we will use a built-in capability of modern web browsers.

At a high level, we want to be able to understand two broad types of queries. Following is the flowchart.

News

We might ask for news. E.g.:

What is the news on Fantastic Beasts in the Guardian?

The bot will query the API of the requested news source (New York Times if none is specified) and summarize the results:

[…] Comparing the first Harry Potter film (2001’s Harry Potter and the Philosopher’s Stone) with the last (2011’s Harry Potter and the Deathly Hallows Part Two) is somewhat akin to comparing Bambi with Reservoir Dogs. We first meet him in 1920s New York — almost 60 years before Harry is even born — where he is […]
(Source: https://www.theguardian.com/books/2016/nov/25/jk-rowling-fantastic-beasts-screenplay)

Knowledge

We might ask a knowledge question. E.g.:

What are RDF triples?

And the bot will answer:

A semantic triple, or simply triple, is the atomic data entity in the Resource Description Framework .\nThis format enables knowledge to be represented in a machine-readable way. Particularly, every part of an RDF triple is individually addressable via unique URIs \u2014 for example, the second statement above might be represented in RDF as http://example.name#BobSmith12 http://xmlns.com/foaf/0.1/knows http://example.name#JohnDoe34
(Source:https://en.wikipedia.org/wiki/Semantic_triple)

How It Works

We define a simple rule to categorize inputs: if the query contains either of the words news or latest, it is a news query. Otherwise, it is a knowledge query.

The predict function of the QueryAnalyzer class is the main entry point for our bot. It performs the above categorization. It calls other functions to

Extract the query and, if applicable, the source from the input
Make necessary API calls
Summarize lengthy content

Finally, it returns the output and a flag indicating if there was any error.

News Queries

We assume input to be of one of the following forms.

What is the latest news on Star Wars in the New York Times?
Read me the latest on Brexit.
Give me news about Marvel Cinematic Universe movies in 2017 from the Guardian.

Parts of Speech and Tags

Noun Chunks

the latest news
Donald Trump
the New York Times

Adpositions? Did You Mean Prepositions?

There is a pattern in sentences structured as above. And prepositions are key.

The topic of search is between the first and the last prepositions. The requested source is at the end after the last preposition. The last noun chunk is the source.

In case a source is not specified, as in the second example, everything after the first preposition is assumed to be the topic of search.

Adpositions, simply put, are prepositions and postpositions.

In a head-initial language like English, adpositions usually precede the noun phrase. E.g. characters from the Marvel Cinematic Universe. While in a head-final language like Gujarati, adpositions follow the noun phrase. These are postpositions. E.g. માર્વેલ ચલચિત્ર જગતના પાત્રો, which translates word by word to: Marvel Cinematic Universe of characters.

Implementation

We invoke get_news_tokens from the QueryExtractor class, which extracts the source and the query from the input. Internally, it calls _split_text to extract noun chunks, parts of speech, and the fully parsed text from the input. We lemmatize terms in the query.

Next, we invoke the get_news function using query on one of the Aggregator classes in media_aggregator.py based on the source. This returns a list of news articles that were sent as a response by the news API. We currently support The Guardian API and The New York Times API.

Finally, we pick the first item (by default) from the response list and summarize it using the shorten_news function.

Knowledge Queries

We assume input to be of one of the following forms.

John Deere
Joan of Arc
Who is Donald Trump?
Who was JRR Tolkien?
What is an RDF triple?
Tell me about particle physics.

Parts of Speech and Tags

Example 1

Example 2

Noun Chunks

Example 1

What
an RDF triple

Example 2

me
he-man
the masters
the universe

Auxiliary Verbs (or Their Absence)

If we find an auxiliary verb, we treat everything after its first occurrence as the query. Thus, in Example 1, the query is RDF triple.

Otherwise, we treat all noun chunks after the first as the query. Thus, in Example 2, the query is he-man the masters the universe.

Implementation

We invoke get_knowledge_tokens from the QueryExtractor class, which extracts the query.

We pass this to the get_gkg function, which queries the Wikipedia API through the wikipedia Python package and returns a 5-sentence summary of the top result.

Summarization

I used the FrequencySummarizer class from Text summarization with NLTK. Alternatively, you could use sumy.

Libraries

In addition to the packages re, bs4, requests, operator, collections, heapq, string and nltk, we use the following.

spaCy: Please set it up as given in the Install spaCy docs. spaCy will help us do some quick NLP. We could use NLTK but spaCy will get you going faster. We use spaCy in this project.
Wikipedia: This helps query the Wikipedia API. You can read the docs of the wikipedia Python package here.
Summarizer: The one I used was borrowed from The Glowing Python blog written by JustGlowing. It summarizes lengthy content. Alternatively, you could use sumy.
Flask-RESTful, Flask (Optional): These are for building a web app and operationalizing our bot through a RESTful web service.

Web App (Optional)

We add a cool webpage from which you can fire off voice queries and have the browser read out the response content. We make use of the Web Speech API for this.

Web Service

We get our Flask-based REST web service up and running in under 20 lines of code. The QueryService class handles requests.

As of now, we only need one service call to send input from our web app to our bot. This is done through the post function of QueryService class. post, in turn, calls the predict function, which is the main entry point as mentioned above.

Website

I built a basic webpage to demonstrate the bot. It uses the Web Speech API to receive voice input and read out content. You can find the index.html file in the templates folder of the project. Make sure you have installed all the required packages and libraries, and that the web service is up and running before you open the website.

Limitations

Our simple bot understands a limited range of requests. It cannot understand other kinds of requests such as follows.

Knowledge requests with a different structure

Explain to me what bootstrap aggregation is.
Tell me something about computational neuroscience.

News requests with a different structure

What does the New York Times say about Roger Federer’s latest match?
What’s happening in the world of tennis?

Knowledge requests of other types

How is cheese made?
Where was JK Rowling born?
Can we build a sky city on Venus?
When did the French Revolution take place?
Why does Jupiter have The Great Red Spot?

Follow-up questions and context

Explain to me what bootstrap aggregation is.

and then: How does it relate to random forests?

Understanding what it refers to in the follow-up question comes under what is known as anaphora resolution. It is all a part of understanding context. Different words mean different things in different contexts. While humans have a nuanced understanding of these, it is significantly more difficult to teach machines the same.

Conclusion and Future Work

We achieved our goal of building a bot based on some rules we defined. We also made use of some NLP techniques. Finally, we deployed our bot onto a web application. However, our bot is limited in the kinds of queries it can understand and answer. Why is its scope of understanding so narrow?

Making computers really understand language is an AI-hard problem. There is a field known as NLU (Natural Language Understanding) within NLP dedicated to this.

Next step is to implement a machine learning-based solution so our bot could potentially understand a much wider range of requests. We will enable it to classify question categories (who, what, when, affirmation, and so on) and fetch information accordingly.