Overview

Goal: To onboard developers onto chatbot the XState-Chatbot code base so that they can modify existing flows or create new ones

Prerequisites

...

.

This document sticks to explaining the chatbot's core features and does not dive into the use cases implemented by the chatbot. There is another document dedicated to it.

Prerequisites

...

• NodeJS

...

• XState

...

SOA

...

Checkout code

...

Install …

1. Recommended IDE: VSCode

2. Postman

...

Run code 

1. using react-app

2. using node.js

3. When to use one vs other

...

Overall philosophy

1. No NL

2. Simple form fill

3. State machine approach

...

Introduction to x-state

1. Required reading ….

...

Developing

1. Introduction to project structure (quick)

   1. Channels

   2. Session manager

   3. Machines

   4. ...

2. Understand current dialog flow - 

   1. Have the new developer interact with existing modules using react-app

   2. How to use visual representation and IDE to understand dialog state including hierarchy / Understand how that flow is provided in the code

   3. General patterns

      1. Hierarchical structure

      2. Typical structure 1 - preprocess -> question -> process -> error or transition

      3. onEntry, onEvent, invoke-onDone, always ... 

      4. When to queue messages vs when to call toUser()

3. Scaffolding

   1. Understand how session manager works

   2. Channels

   3. envconfig

   4. Telemetry

   5. Localization

   6. Handling location - Google APIs

   7. What is Postgres used for

   8. What is ElasticSearch used for. Sanctifying user info

4. Best practices

   1. Naming convention

   2. Testing

   3. How to handle large lists

      1. Escape to the web, e.g., city & locality

      2. Two-level structure, e.g., complaint type

   4. Always monitor DROP OFF / COMPLETION for three months when introducing new dialog in the field

5. Future thoughts

...

Developer - Test

1. Changing prompt and response

2. Modifying dialog flow

   1. Rearranging questions

   2. Adding a new question

   3. Adding a conditional question 

3. Developing new dialog

4. Build ELK dashboard to understand drop off funnel

...

Operations

1. What do we have deployed today. Systems diagram ….

2. ELK

Initial Setup

1. Git - In macOS, most probably, it comes preinstalled. Git Installation Guide

2. Visual Studio Code (VSCode) - Great for Node JS Project Management

3. Install NodeJS

   1. If you have HomeBrew, an easier way to install would be “brew install node”

   2. After installing you should be able to run

      1. node -v

      2. npm -v

4. GitHub Repository - XState-Chatbot

   1. Open this folder in the VSCode

   2. Terminal could be opened within the VSCode (Ctrl + `)

   3. For the first time installing dependencies: “npm install”

   4. To run the app: “npm start”

5. To run the nodejs-server version of the code:

   1. brew install cask postgresql

   2. Install PostgreSQL (username - postgres, password - postgres, host - localhost, port - 5432). If the values differ, please modify those in the app.

   3. Create a database named “chat”

   4. Run the .sql files present in /db/migration/main folder

NodeJS Project Convention

1. All file and folder names in kebab-case

2. Class names in PascalCase. Variable names in camelCase

Development Path

...

Dummy service

...

eGov service

...

In-Memory database

...

react (1) - initial dialog dev, external developers

nodejs

...

react (2) - dialog hardening and testing

nodejs

...

Postgres database 

(NA for React)

...

nodejs

...

nodejs (3) - staging and production

Message Format

...

Text

1. User input

2. Location

   1. Geocode (lat, long) pair

   2. Street address

Binary

...

Image

...

• PostgreSQL

• Kafka(optional)

Key Functionalities

• Build a chat flow to facilitate a user to interact with rainmaker modules

• Link a chat flow with backend services

Deployment Details

1. Deploy the latest version of xstate-chatbot

2. Configure /xstate-chatbot API’s to be a whitelisted open endpoint in zuul. There are 3 API’s /message, /status and /reminder. The postman collection is present in this document

3. Add indexer-config to the egov-indexer to index all the telemetry messages

Other configuration details are mentioned as part of the XState-Chatbot Integration Document.

Overall Philosophy of Chatbot

This chatbot solves the basic form filling aspect of a chat flow. By collecting the information from the user, an API call can be made to the rainmaker backend services to fulfill what the user wants to do. It uses the concept of StateCharts (similar to State Machines) to maintain the state of the user in a chat flow and store the information provided by the user. XState is a JavaScript implementation of StateCharts. All chat flows are coded inside the XState framework.

Basic Introduction to XState

XState is a JavaScript implementation of StateCharts. There is detailed documentation available to study XState. Few of the concepts of XState that are used in Chatbot are listed below. Basic knowledge of these concepts is necessary. It can also be learned while going through the chat flow implementation of pilot use cases of PGR and Bills.

1. State

   1. State Nodes

   2. Hierarchical State Nodes

2. Actions

   1. onEntry

   2. invoke…onDone…onError

3. Guard Conditions

Few tips about using XState. These have been followed throughout the pilot chat flows.

1. If we want to move to any state which is not at the same hierarchical level, then we should assign it a unique id value. If it has an id value, we can address it using the # qualifier in the target attribute.

2. As id should be unique, please make sure there aren’t multiple states with the same id value. If there is a duplicate, the machine won’t function as expected.

3. Any actions(like onEntry) should be surrounded by assign.

   Code Block
   assign( (context, event) => { ... } )

   This would include almost all functions except the guard condition code snippets.

Understanding the Chat Flow

All the interactions with the user - sending a message to the user and processing an incoming message from the user is coded as a state in the State Machine. It would be a nice start to test any chat flow with the supplementary react-app provided for the developers to execute the state machine locally. (Please follow the guidelines in the README of the react-app.)

We have followed few standard patterns to code any chat interaction. Please try to follow these patterns to code any new chat flow. These patterns are explained below. You can also study those by browsing through the code of the pilot use cases of PGR and Bills.

1. The chat states would only include dialog-specific code. Any code related to backend service should be written as a part of a separate …-service.js file.

2. Any code that doesn’t include any asynchronous API call can be written as a part of the onEntry function or an action.

3. If the function needs to make an API call, that would have to be written with the invoke-onDone pattern. The asynchronous function should be written as a part of the service file. The consolidated data returned by it can be processed in the state of the dialog file.

4. Helper functions are written in dialog.js file. It is advised to use those functions as much as possible rather than writing any custom logic in dialog files.

Scaffolding

Apart from the chat flow and its backend service API calls, few other components are present in the project. These components do NOT need to be modified to code any new chat flow or changing an existing chat flow. These components with a short description for each are listed below:

1. Session Manager: It manages sessions of all the users on a server. It will store the user’s state in a datastore, update it, and read it when any new message is received on the server. Based on the state of the user, it will create a state machine and send the incoming message event to the state machine. It sanctifies the state (any sensitive data like the name and mobile number of a user are removed) before storing the state to the datastore.

2. Repository: It is the datastore where the states of the users get stored. To reduce dependency, an in-memory repository is also provided, which can be used by configuring an environment variable. So to run the chatbot service, PostgreSQL isn’t a hard dependency, but it is advisable to use the PostgreSQL repo provider.

3. Channel Provider: There can be many different WhatsApp Providers. Any one of the providers will be configured to be used. A separate console WhatsApp Provider is present for the developer to test the chatbot server locally. Postman collection to mimic receiving messages from a user to the server is present in the project directory.

4. Localization: Every message to be sent to the user is stored within the chatbot. Localization service is not being used. These messages are present near the bottom of the dialog files. A separate localization-service.js is provided to get the messages for the localization codes for the messages that are not owned by the chatbot. For example, the PGR complaint types data is under the ownership of the PGR module, and the messages for such can be fetched from the egov-localization-service using the functions provided in the localization-service.js.

5. Service Provider: To ease the initial dialog development, instead of the coding API calls to the backend services, we can configure the chat flow to use a dummy service. This can be configured using an environment variable and modifying the service-loader.js file.

6. Telemetry: Chatbot logs telemetry events to a kafka topic. (Any sensitive data will get masked before indexing the events onto ElasticSearch by egov-indexer.) Following events get logged:

   1. Incoming message

   2. Outgoing message

   3. Transition of state