Your Voice Assistant in ReactNative & DialogFlow

Not long ago the developers were building monolithic applications, which later were divided into client and server code using APIS. Then we reworked the back-end on micro-services to have more scalability and reliability. 
Someone recently, in a meeting I was, named this term as Technology Waves. Now you can think now how many of them you lived through the years of experience.

I recently joined a new team where we build prototypes for testing concepts and ideas to learn from. We sometimes use end to end testing in devices to have clearer insights.

Nowadays the world lives the Technology Wave of Artificial Intelligence and the Bots and we didn’t want to linger behind. We started to think about something simple that can be plugged in some different devices.

A simple, flexible and portable proof of concept came to us and can be used for many purposes. 
The idea is to use a Natural Language Processor (NLP), that can be requested via HTTP API from a client App. 
The constructions would be something like this, the NLP will answer our requests. The client App needs to recognise user’s voice, transforms it into text which then queries the NLP service, wait for the response and convert the callback text into voice.

This experiment will allow us to see how is the user voice interaction with our App and get early feedback from our users like intent recognition issues, noise problems with microphones, users expectations with the technology, and some others.

Let’s navigate through the parts in detail.

1. The NLP Service

We will try DialogFlow which is an NLP service from Google, it has many integrations, SDKs for many languages and prebuilt agents. It works very straightforward with Google Assistant.

Before explaining further let’s review some concepts. An intent is a pattern detected by the NLP service using what is called utterances which are an array of sentences that defines a behaviour model. Normally those intents are keywords that you can accommodate to easy your system.

The NLP service defines some intents and some utterances which will be used for the service to respond our requests.The service also allows queries via API, and that is what we need for the PoC.

Intent Menu

New intents can be added. Let’s add a new one.

Intent creation Menu.

There is a side console where you can test your intents manually. And following the docs, the NLP service can be requested through HTTP.

2. The Client

As mentioned earlier sometimes we use end-to-end testing in devices. Being portable is always good when building prototypes so React-Native fit our needs. We like to test this out with Android over iOS, because of testing available devices.
This time we assume you have already installed react-native-cli and started an Android project.
Otherwise, in this, you can find the steps to carry on.

Firstly we are going to assemble the modules:
For voice recognition, we go for this module,

For text to speech, we go for this module,

Then we ended up with some of the config files changed:


3. Last step

Once all the set up is finished let see how integrated into the android.index.js

With not many lines we have a customise assistant in Android. But as expected not everything works at first glance.

4. Potential issues and strong points

Now that we have built our own assistant in Android, very flexible to customise in terms of intents and response with a web interface, it is time for some takeaways.

a. Sometimes the cache is playing against you when debugging your app. External dependencies are a bit tricky to add plus the NPM cache could play against us. Keeping on eye on that and reinstalling the dependencies should make it work.

b. DialogFlow SDK: I decided for implementing the query fetching myself cause SDK’s are not working straightforward with React-Native. Documentation is not excellent but the HTML+JS is plenty for our demo.

c. Remote debugging is a key point. With simple console.log, console.warn and console.error, you can debug what is going on in your application plus the browser connection which it helps you track what your application is doing. For example in this demo is important to know how request and response are doing in terms of the application life, for that you can take a look how the voice is recognised in the request and the response.

React-Native remote debug

d. Dialogflow is awesome, clear interface, easy to use, available in many languages and flexible enough to plug your own fulfilment.

If you like to have a deeper look here is the code of the demo

Hope you like this, if so, you can spread it.

Thanks for reading.