W&S : Privacy Changes Backend Technical Documentation
- Hina Makhija
Overview
As digitisation is increasing, privacy has become one of the major requirements in the digital world. Masking the PII data is one such ways to achieve privacy.
This document highlights the changes need to be done in a Water and Sewerage module to support the privacy feature which will mask the PII data during search and updates at different levels.
Pre-requisites
Prior Knowledge of Java/J2EE.
Prior Knowledge of Spring Boot.
MDMS Service
Encryption Service
MDMS changes
Update the SecurityPolicy.json file.
The Security Policy mdms file, must have the model configuration for fields to be encrypted/decrypted.
Following models have been used for W&S in SecurityPolicy.json file:
{
"model": "WnSConnection",
"uniqueIdentifier": {
"name": "uuid",
"jsonPath": "/connectionHolders/0/uuid"
},
"attributes": [
{
"name": "ownerType",
"jsonPath": "connectionHolders/*/ownerType",
"patternId": "005",
"defaultVisibility": "PLAIN"
},
{
"name": "relationship",
"jsonPath": "connectionHolders/*/relationship",
"patternId": "005",
"defaultVisibility": "PLAIN"
}
],
"roleBasedDecryptionPolicy": [
{
"roles": ["WS_CEMP","WS_DOC_VERIFIER","WS_FIELD_INSPECTOR","WS_APPROVER","WS_CLERK","SW_CEMP","SW_DOC_VERIFIER","SW_FIELD_INSPECTOR","SW_APPROVER","SW_CLERK"],
"attributeAccessList": [
{
"attribute": "ownerType",
"firstLevelVisibility": "MASKED",
"secondLevelVisibility": "PLAIN"
},
{
"attribute": "relationship",
"firstLevelVisibility": "MASKED",
"secondLevelVisibility": "PLAIN"
}
]
},
{
"roles": ["REINDEXING_ROLE"],
"attributeAccessList": [
{
"attribute": "ownerType",
"firstLevelVisibility": "ENCRYPTED",
"secondLevelVisibility": "PLAIN"
},
{
"attribute": "relationship",
"firstLevelVisibility": "ENCRYPTED",
"secondLevelVisibility": "PLAIN"
}
]
}
]
},
{
"model": "WnSConnectionOwner",
"uniqueIdentifier": {
"name": "uuid",
"jsonPath": "/uuid"
},
"attributes": [
{
"name": "connectionHoldersMobileNumber",
"jsonPath": "mobileNumber",
"patternId": "001",
"defaultVisibility": "PLAIN"
},
{
"name": "fatherOrHusbandName",
"jsonPath": "fatherOrHusbandName",
"patternId": "002",
"defaultVisibility": "PLAIN"
},
{
"name": "gender",
"jsonPath": "gender",
"patternId": "005",
"defaultVisibility": "PLAIN"
},
{
"name": "correspondenceAddress",
"jsonPath": "correspondenceAddress",
"patternId": "005",
"defaultVisibility": "PLAIN"
},
{
"name": "ownerType",
"jsonPath": "connectionHolders/*/ownerType",
"patternId": "005",
"defaultVisibility": "PLAIN"
},
{
"name": "plumberInfoMobileNumber",
"jsonPath": "plumberInfo/*/mobileNumber",
"patternId": "001",
"defaultVisibility": "PLAIN"
},
{
"name": "assigneeMobileNumber",
"jsonPath": "assignes/*/mobileNumber",
"patternId": "001",
"defaultVisibility": "PLAIN"
},
{
"name": "relationship",
"jsonPath": "connectionHolders/*/relationship",
"patternId": "005",
"defaultVisibility": "PLAIN"
}
],
"roleBasedDecryptionPolicy": [
{
"roles": ["WS_CEMP","WS_DOC_VERIFIER","WS_FIELD_INSPECTOR","WS_APPROVER","WS_CLERK","SW_CEMP","SW_DOC_VERIFIER","SW_FIELD_INSPECTOR","SW_APPROVER","SW_CLERK"],
"attributeAccessList": [
{
"attribute": "connectionHoldersMobileNumber",
"firstLevelVisibility": "MASKED",
"secondLevelVisibility": "PLAIN"
},
{
"attribute": "gender",
"firstLevelVisibility": "MASKED",
"secondLevelVisibility": "PLAIN"
},
{
"attribute": "fatherOrHusbandName",
"firstLevelVisibility": "MASKED",
"secondLevelVisibility": "PLAIN"
},
{
"attribute": "ownerType",
"firstLevelVisibility": "MASKED",
"secondLevelVisibility": "PLAIN"
},
{
"attribute": "correspondenceAddress",
"firstLevelVisibility": "MASKED",
"secondLevelVisibility": "PLAIN"
},
{
"attribute": "relationship",
"firstLevelVisibility": "MASKED",
"secondLevelVisibility": "PLAIN"
},
{
"attribute": "plumberInfoMobileNumber",
"firstLevelVisibility": "MASKED",
"secondLevelVisibility": "PLAIN"
},
{
"attribute": "assigneeMobileNumber",
"firstLevelVisibility": "MASKED",
"secondLevelVisibility": "PLAIN"
}
]
},
{
"roles": ["REINDEXING_ROLE"],
"attributeAccessList": [
{
"attribute": "connectionHoldersMobileNumber",
"firstLevelVisibility": "ENCRYPTED",
"secondLevelVisibility": "PLAIN"
},
{
"attribute": "gender",
"firstLevelVisibility": "ENCRYPTED",
"secondLevelVisibility": "PLAIN"
},
{
"attribute": "fatherOrHusbandName",
"firstLevelVisibility": "ENCRYPTED",
"secondLevelVisibility": "PLAIN"
},
{
"attribute": "ownerType",
"firstLevelVisibility": "ENCRYPTED",
"secondLevelVisibility": "PLAIN"
},
{
"attribute": "correspondenceAddress",
"firstLevelVisibility": "ENCRYPTED",
"secondLevelVisibility": "PLAIN"
},
{
"attribute": "relationship",
"firstLevelVisibility": "ENCRYPTED",
"secondLevelVisibility": "PLAIN"
},
{
"attribute": "plumberInfoMobileNumber",
"firstLevelVisibility": "ENCRYPTED",
"secondLevelVisibility": "PLAIN"
},
{
"attribute": "assigneeMobileNumber",
"firstLevelVisibility": "ENCRYPTED",
"secondLevelVisibility": "PLAIN"
}
]
}
]
},
{
"model": "WnSConnectionDecrypDisabled",
"uniqueIdentifier": {
"name": "uuid",
"jsonPath": "/connectionHolders/0/uuid"
},
"attributes": [
{
"name": "ownerType",
"jsonPath": "connectionHolders/*/ownerType",
"patternId": null,
"defaultVisibility": "PLAIN"
},
{
"name": "relationship",
"jsonPath": "connectionHolders/*/relationship",
"patternId": null,
"defaultVisibility": "PLAIN"
}
],
"roleBasedDecryptionPolicy": []
},
{
"model": "WnSConnectionOwnerDecrypDisabled",
"uniqueIdentifier": {
"name": "uuid",
"jsonPath": "/uuid"
},
"attributes": [
{
"name": "connectionHoldersMobileNumber",
"jsonPath": "mobileNumber",
"patternId": null,
"defaultVisibility": "PLAIN"
},
{
"name": "fatherOrHusbandName",
"jsonPath": "fatherOrHusbandName",
"patternId": null,
"defaultVisibility": "PLAIN"
},
{
"name": "gender",
"jsonPath": "gender",
"patternId": null,
"defaultVisibility": "PLAIN"
},
{
"name": "correspondenceAddress",
"jsonPath": "correspondenceAddress",
"patternId": null,
"defaultVisibility": "PLAIN"
},
{
"name": "ownerType",
"jsonPath": "connectionHolders/*/ownerType",
"patternId": null,
"defaultVisibility": "PLAIN"
},
{
"name": "plumberInfoMobileNumber",
"jsonPath": "plumberInfo/*/mobileNumber",
"patternId": null,
"defaultVisibility": "PLAIN"
},
{
"name": "assigneeMobileNumber",
"jsonPath": "assignes/*/mobileNumber",
"patternId": null,
"defaultVisibility": "PLAIN"
},
{
"name": "relationship",
"jsonPath": "connectionHolders/*/relationship",
"patternId": "005",
"defaultVisibility": "PLAIN"
}
],
"roleBasedDecryptionPolicy": []
},
{
"model": "WnSConnectionPlumber",
"uniqueIdentifier": {
"name": "applicationNo",
"jsonPath": "/applicationNo"
},
"attributes": [
{
"name": "plumberInfoMobileNumber",
"jsonPath": "plumberInfo/*/mobileNumber",
"patternId": "001",
"defaultVisibility": "PLAIN"
}
],
"roleBasedDecryptionPolicy": [
{
"roles": ["WS_CEMP","WS_DOC_VERIFIER","WS_FIELD_INSPECTOR","WS_APPROVER","WS_CLERK","SW_CEMP","SW_DOC_VERIFIER","SW_FIELD_INSPECTOR","SW_APPROVER","SW_CLERK"],
"attributeAccessList": [
{
"attribute": "plumberInfoMobileNumber",
"firstLevelVisibility": "MASKED",
"secondLevelVisibility": "PLAIN"
}
]
},
{
"roles": ["REINDEXING_ROLE"],
"attributeAccessList": [
{
"attribute": "plumberInfoMobileNumber",
"firstLevelVisibility": "ENCRYPTED",
"secondLevelVisibility": "PLAIN"
}
]
}
]
},
{
"model": "WnSConnectionPlumberDecrypDisabled",
"uniqueIdentifier": {
"name": "applicationNo",
"jsonPath": "/applicationNo"
},
"attributes": [
{
"name": "plumberInfoMobileNumber",
"jsonPath": "plumberInfo/*/mobileNumber",
"patternId": null,
"defaultVisibility": "PLAIN"
}
],
"roleBasedDecryptionPolicy": []
}Also add following under roleBasedDecryptionPolicy of User model(already existing):
"roleBasedDecryptionPolicy": [
{
"roles": ["REINDEXING_ROLE"],
"attributeAccessList": [
{
"attribute": "mobileNumber",
"firstLevelVisibility": "ENCRYPTED",
"secondLevelVisibility": "PLAIN"
}
]
},
{
"roles": ["WS_CEMP","WS_DOC_VERIFIER","WS_FIELD_INSPECTOR","WS_APPROVER","WS_CLERK","SW_CEMP","SW_DOC_VERIFIER","SW_FIELD_INSPECTOR","SW_APPROVER","SW_CLERK"
],
"attributeAccessList": [
{
"attribute": "mobileNumber",
"firstLevelVisibility": "MASKED",
"secondLevelVisibility": "PLAIN"
},
{
"attribute": "fatherOrHusbandName",
"firstLevelVisibility": "MASKED",
"secondLevelVisibility": "PLAIN"
},
{
"attribute": "correspondenceAddress",
"firstLevelVisibility": "MASKED",
"secondLevelVisibility": "PLAIN"
},
{
"attribute": "name",
"firstLevelVisibility": "PLAIN",
"secondLevelVisibility": "PLAIN"
},
{
"attribute": "emailId",
"firstLevelVisibility": "MASKED",
"secondLevelVisibility": "PLAIN"
},
{
"attribute": "permanentAddress",
"firstLevelVisibility": "MASKED",
"secondLevelVisibility": "PLAIN"
},
{
"attribute": "guardian",
"firstLevelVisibility": "MASKED",
"secondLevelVisibility": "PLAIN"
}
]
}
] Refer the SecurityPolicy.json file for full reference.
For modules, there is no such hard rule or pattern for naming the model, but it should be related to that service.
ex: 1.) For user service we have two security policy model User which is used when a user tries to search other user data and he/she will get the PII data in masked, plain or encrypted form depending on the visibility sets for an attribute in the mdms.
And another model is UserSelf which is used when a user tries to search its own data and he/she will get the data as per the configuration set there. For report and searcher config the model name should be similar to the value that we are setting in the field decryptionPathId. ex:- Employee Report. Employee Report Security Model
2.) For W&S we have models WnSConnection and WnSConnectionOwner which are for ConnectionHolderDetails.
We were required to have 2 different models for connectionHolder because some data for connectionHolder comes from W&S tables and some comes from User table. So to maintain the consistency we had to have 2 different models.
For an attribute where its firstLevelVisibility is set as "Masked" and whenever the respective search API is called without the plain Access Request then in the API response for that attribute we will get the masked value
for ex:- if for mobile number attribute’s firstLevelVisibility is masked and its plain value is 9089243280 then in response, we will get value as ******3280 and the masking pattern is defined in the MaskingPattern mdms file and the pattern is picked up based on patternId. Similarly, if firstLevelVisibility is set as "ENCRYPTED" we will get the encrypted value of that plain data (which is present in DB) in the response.
NOTE: For adding of new attribute for encryption, following things need to be kept in mind:
We do not have a direct approach to it, but a workaround as follows:
We need to make sure which property has to be encrypted and what is the path of the property in the Request/Response object of W&S. If PII data is for connectionholder and is coming from WnS tables directly, then with Proper name and path of object we can add a new Property in existing model WnSConnection.
Inclusion of any new attribute here would need encryption of the old data for this new property.
For that, in MDMS, we will have to replace the existing model attributes with the only new attributes and hit the_encryptOldDataapi. Once old data encryption is done, we can put back all the required attributes (old +new) in the model.
Also, before starting the encryption of old data, we will have to check the latest record of tableeg_ws_enc_audit/eg_sw_enc_audit/eg_pt_enc_audit(for PT) . If the latest record has offset and record count value other than 0 then insert a random record with offset and record count as 0 andcreatedtimeandencryptiontimeas current timestamp in millis in utc.Any data other than that of connectionHolder would need a new model to be created and changes at code level as well.
For old data encryption of new property other than that of connectionHolder, we will have to have code changes as well.
Backend service changes
Update the pom.xml file.
Upgrade services-common library version.
<dependency> <groupId>org.egov.services</groupId> <artifactId>services-common</artifactId> <version>1.1.0-SNAPSHOT</version> </dependency>Upgrade tracer library version.
Add enc-client library
Application Properties changes:
In ws-services encryption and decryption end points should be declared as follows:
In sw-services encryption and decryption end points should be declared as follows:
EncryptionDecryptionUtil.java:
We need an interfacing file for handling the encryption and decryption of attributes and for interacting with enc-client library directly.
For reference follow the below code snippet
You may find the file here in the repo.
UnmaskingUtil.java:
UnmaskingUtil.java file helps in extracting the plain data of Owner from the User service.
This file also contains a method that swaps the PlumberInfo in the similar manner
For reference follow the below code snippet
You may find the file here in the repo.
Important methods to be added:
Method to encrypt the water/sewerage requestBody data:
Following method is added in WaterServiceImpl.java for encrypting the entire data
Method to decrypt the water/sewerage data:
Following method is added in WaterServiceImpl.java for decrypting the entire data
_search api changes:
During search call the PII data needs to be masked in response received.
For this the searchCriteria needs to be encrypted before fetching details and eventually the data retrieved needs to be decrypted and then returned to the user.
Following changes need to be made in WaterServiceImpl.java
You may find the file here in the repo.
If you want to get all the PII data in plain format then in the search request add the search param “isInternalCall“ as “true”
If the user wants only specific fields to be unmasked then add the plainAccessRequest in the RequestBody in the following format:
plainAccessRequest contains :
1. recordId which will take uuid of the user as an input for which PII data has to be unmasked, and
2. plainRequestFields will take and array of attributes that need to be unmasked. These arrtributes should comply with the attributes used in the Mdms SecurityPolicy.json’s models created.
_create and _update api changes:
In _create and _update api calls the data needs to be encrypted before the data for application/connection is being pushed to respective topics.
A sample codeline for encrypting data in these calls is as follows:
A sample codeline for decrypting data in these calls is as follows:
Entire method for create call can be found here: create method
and update call can be found here: update method
Encryption of Old Data in the dB :
Before using the privacy feature in any environment, first the encryption of existing data in the dB should be done.
An api for the same is written in the service and needs to be triggered by port-forwarding the respective service pod.
The data encryption api uses the existing plainSearch method and the encryption shall take place tenantId-wise. If tenantId is not specified then all the tenants are picked from the mdms repository and the encyption happens for all the tenants. However, if the tenantId is specified, then the encryption happens only for that tenantId.
Following method is added to execute the oldDataEncryption
In WaterController.java
In WaterEncryptionService.java
This service is solely for old data encryption and can be referred from here:
https://github.com/egovernments/DIGIT-Dev/blob/8ec8592b73470c6dcdb1508caa1e6b1cf8ebb2ca/municipal-services/ws-services/src/main/java/org/egov/waterconnection/service/WaterEncryptionService.java
A new persister file needs to be added for keeping track of progress of oldDataEncryption.
Refer this ws-enc-audit-persister.yml and sw-enc-audit-persister.yml for the same.
Here is the Postman collection for all property, water and sewerage services for old Data encryption :
https://www.getpostman.com/collections/b3858ec2020462a6407d
Re-indexing of Old Data in the base indexes:
There is NO need of re-indexing the base indexes(viz. water-services/ sewerage-services) once the api for old data encryption is executed as this will happen during the api execution itself.
For any new data getting created, the new data will get saved in the encrypted format only in the indexes.
There are some changes in the Indexer file that need to be made for W&S to get the encrypted data from the external service like Property-service.
So, in the URL mentioned above in apiRequest, change the role from EMPLOYEE to REINDEXING_ROLE.
Reference for changes :
[UM-4382]- Updated indexer for water service by hinamakhija-eGov · Pull Request #2342 · egovernments/configs
[UM-4382]-Updated indexer for sewerage by hinamakhija-eGov · Pull Request #2345 · egovernments/configs
This role is newly defined role in MDMS as already mentioned in the changes on the top.
This Role helps in getting the Encrypted data as per the firstLevelVisibilty defined for it.
For W&S, we need encrypted MobileNumber in the index because W&S inbox uses Elastic search and we require mobileNumber search for the same hence owner MobileNumber has to be there mandatorily in the index for search.
The changes need to be made in all the topics for W&S indexes. For more details find the changes here.
Enabling / Disabling Privacy:
To enable or disable decryption in privacy, we need to make change only privacy variable present in the environment file in DevOps.
For enabling, its value should be “true” else “false”.
For water-services variable is as follows:
For sewerage-services:
For user-service:
The variable can be found in the file here
For detailed steps to configure the above changes refer document: Detailed Steps to configure privacy in W&S module