Backend:

Overview

DSS has two sides to it. One being the process in which the Data is pooled to ElasticSearch and the other being the way it is fetched, aggregated, computed, transformed and sent across.
As this revolves around a variety of Data Set, there is a need for making this configurable. So that, tomorrow, given a new scenario is introduced, then it is just a configuration away from getting the newly introduced scenario involved in this flow of process. 

This document explains the steps on how to define the configurations for both sides of DSS. Analytics and Ingest Pipeline Services. 

Abbreviations

Ingest  : Micro Service which runs as a pipeline and manages to validate, transform and enrich the incoming data and pushes the same to ElasticSearch Index

Analytics : Micro Service which is responsible for building, fetching, aggregating and computing the Data on ElasticSearch to a consumable Data Response. Which shall be later used for visualizations and graphical representations. 

JOLT  : JSON to JSON transformation library written in Java where the "specification" for the transform is itself a JSON document

Modules / Domain Level : These are the Services in this context. Each of the services, such as Property Tax, Trade License, Water and Sewerages are considered as Modules / Domains

Chart : Each individual graphical representation is considered as Chart in specific. For example, a Metric of Total Collection is considered as a Chart. 

Visualization : Group of different Charts is considered as a Visualization. For example, the group of Total Collection, Target Collection and Target Achieved, is considered as a Metric Collection of Charts and thus it becomes a Visualization.  

Ingest Pipeline Configurations

Below are list of configurations

1. Topic Context Configurations

2. Validator Schema

3. JOLT Transformation Schema

4. Enrichment Domain Configuration 

5. JOLT Domain Transformation Schema

Descriptions :-

1. Topic Context Configurations

Topic Context Configuration is an outline to define which data is received on which Kafka Topic. 

Indexer Service and many other services are sending out data on different Kafka Topics. If the Ingest Service is asked to receive those data and pass it through the pipeline, the context and the version of the data being received has to be set. This configuration is used to identify as in which kafka topic consumed the data and what is the mapping for that. 

Open

Figure 1.

Click here for Full Configuration

2. Validator Schema

Validator Schema is a configuration Schema Library from Everit
By passing the data against this schema, it ensures whether the data abides by the rules and requirements of the schema which has been defined. 

Open

Figure 2.

Click here for an example configuration

3. JOLT Transformation Schema

JOLT is a JSON to JSON Transformation Library. In order to change the structure of the data and transform it in a generic way, JOLT has been used. 

While the transformation schemas are written for each Data Context, the data is transformed against the schema to obtain a transformed data. 

Follow the slide deck for JOLT Transformations

Figure 3.

Click here for an example configuration

4. Enrichment Domain Configuration 

This configuration defines and directs the Enrichment Process which the data goes through. 

For example, if the Data which is incoming is belonging to a Collection Module data, then the Collection Domain Config is picked. And the based on the Business Type specified in the data, the right config is picked. 

In order to enhance the data of Collection, the domain index specified in the configuration is queried with the right arguments and the response data is obtained, transformed and set. 

Click here for an example configuration

5. JOLT Domain Transformation Schema

As a part of Enhancement, once the domain level object is obtained, we might not need the complete document as is in the end data product. 

Only those parameters which should be or can be used for an aggregation and representation are to be held and others are to be discarded. 

In order to do that, we make use of JOLT again and write schemas to keep the required ones and discard the unwanted ones. 

The above configuration is used to transform the data response in the enrichment layer. 

Figure 4.

Click here for an example configuration

6.  Use case:- JOLT Transformation Schema for collection V2

 JOLT transformation schema for payment-v1 has taken as a use case to explain the context  collection and context version v2. The payment records are processed/transformed with the schema. The schema supports splitting the billing records in a independent new record. So if there are 2 bill items in the collection/payment incoming data then this results into 2 collection records in turn.

Figure 5.

Click here for an example configuration

Here: $i, the variable value that gets incremented for the number of records of paymentDetails 

$j, the variable value that gets incremented for the number of records of billDetails.

Note: For kafka connect to work, Ingest pipeline application properties or in environments direct push must be disabled.

es.push.direct=false

Analytics Configurations 

Below are list of configurations

1. Chart API Configuration

2. Master Dashboard Configuration

3. Role Dashboard Mappings Configuration

Description :

1. Chart API Configuration

Each Visualization has their own properties. Each Visualization comes from different data sources (Sometimes it is a combination of different data sources) 

In order to configure each visualization and their properties, we have Chart API Configuration Document. 

In this, Visualization Code, which happens to be the key, will be having its properties configured as a part of configuration and are easily changeable.

Figure 6

Click here for an example configuration

2. Master Dashboard Configuration

Master Dashboard Configuration is the main configuration which defines as which are the Dashboards which are to be painted on screen. 

It includes all the Visualizations, their groups, the charts which comes within them and even their dimensions as what should be their height and width. 

Figure 7.

Click here for an example configuration

3. Role Dashboard Mappings Configuration

Master Dashboard Configuration which was explained earlier hold the list of Dashboards which are available.
Given the instance where Role Action Mapping is not maintained in the Application Service, this configuration will act as Role - Dashboard Mapping Configuration 

In this, each Role is mapped against the Dashboard which they are authorized to see

This was used earlier when the Role Action Mapping of eGov was not integrated.
Later, when the Role Action Mapping started controlling the Dashboards to be seen on the client side, this configuration is just used to enable the Dashboards for viewing. 

Figure 8.

Click here for an example configuration

Adding Configurations

1. Adding Roles and Dashboards :

Figure 9.

To add new dashboard, MasterDashboardConfig.json (dashboards node) has to be modified as below in Figure 10.

Note: dashboards array add a new dashboard as given below

Adding Visualizations in existing Dashboard

To add new visualisations, again MasterDashboardConfig.json (vizArray node) has to be modified as below as shown in Figure 11.

Note: vizArray is to hold multiple visualizations

 Adding charts for visualizations :

To add new chart, chartApiConf.json has to be modified as shown below.  A new chartid has to be added with chart node object.

Metric chart Sample as shown in Figure 12.

Pie  chart Sample as shown in Figure 13.

Figure 13.

Line chart Sample as shown in Figure 14.

Figure 14.

Table  chart Sample: This chart comes with 2 kind, table and xtable.

table (as shown in Figure 15.) type allows to added aggregated fields added as available in the query keys, hence to extract the values based on the key, aggegationPaths needs to add along with their data type as in pathDataTypeMapping.

xtable(as shown in Figure 16.) type allows to add multiple computed fields with the aggregated fields dynamically added.

To add multiple computed columns, computedFields [] where actionName (IComputedField<T> interface), fields [] names as in exist in query key, newField as name to appear for computation must be defined.

https://github.com/egovernments/configs/blob/master/egov-dss-dashboards/dashboard-analytics/ChartApiConfig.json for the full configuration in detail.

Steps to create a charts and visualise are:

1. Create/Add a chart in chartApiConf.json

2. Add a visualization for existing dashboard in MasterDashboardConfig.json  as defined above. 

3. Or in order to create/add  a new dashboard create dashboard in MasterDashboardConfig.json and create a role in RoleDashboardConfig.json

Configuration Changes for DrillThrough :

 1. Example Drill through in Ward table in Property Dashboard.

wardDrillDown is the visualization code for PT Drill Down. kind is the attribute that shows the type of visualization code. Apart from two things all the attributes are common.

 2. Example Drill through in ComplaintList table in PGR Dashboard.

complaintDrillDown is the visualization code for PGR Drill Down.

The above complaintDrillDown visualization code called in the drill chart parameter.