Ifix-Data-Migration
- Debasish Chakraborty
The functionality of this task is to to migrate the Master data and the fiscal event from Mongo-db to Postgres
STEP 1: EXISTING DATA MIGRATION FROM MONGO TO IFIX
We have to migrate the following collection from mongo to postgres.
From MGRAMSEVA NAMESPACE MONGO.
COLLECTION NAME IN MGRAMSEVA MONGO | TABLE NAME IN MGRAMSEVA POSTGRES |
|---|---|
department | department |
expenditure | expenditure |
project | project |
departmentEntity | department_entity |
departmentHierarchyLevel | department_hierarchy_level |
departmentEntity(children details) | department_entity_children (this have mapping of parent-child of department entity) |
FROM IFIX NAMESPAECE MONGO
COLLECTION NAME IN IFIX MONGO | TABLE NAME IN IFIX POSTGRES |
|---|---|
chartOfAccount | eg_ifix_chartofaccount |
fiscal_event | eg_ifix_fiscal_event |
fiscal_event(amount details) | eg_ifix_amount_detail |
fiscal_event(reciever details) | eg_ifix_receivers |
1. We have two fix-migration-toolkit-service running inn IFIX and Mgramseva
ifix-migration-toolkit : build “ifix-migration-toolkit:develop-2557812-9 “ in mgramseva namespace
ifix-migration-toolkit : build “egovio/ifix-migration-toolkit:develop-b7a2050-13” in Ifix namespace
Git repo: https://github.com/egovernments/iFix-Dev/tree/develop/utilities/ifix-migration-toolkit
The functionality of this service is to call the plain search api of concerned service that connnects to mongo db of those service and fetch the result and migrate those to postgres db. We have four APIs to migrate data from mongo to postgres
https://api.postman.com/collections/24408626-b519a61e-034c-4f3f-b4ee-7829dc9e1b67?access_key=PMAT-01H8H1MB52HJ0FY3MQQ9FXA73E
2. Services and branch details you need to deploy to get the db details from Mongo db.
a. Ifix-Department-entity-service in mgramseva namespace:
Branch and git repo: https://github.com/egovernments/iFix-Dev/tree/mongodb-plainsearch/reference-adapter/ifix-department-entity-service
b. Adapter-master-data-service in mgramseva namespace:
Branch and git repo: https://github.com/egovernments/iFix-Dev/tree/mongodb-plainsearch/reference-adapter/adapter-master-data-service
c. Ifix-master-data-service in Ifix namespace:
Branch and git repo: https://github.com/egovernments/iFix-Dev/tree/mongodb-plainsearch/domain-services/ifix-master-data-service
d. Fiscal-event-service in Ifix namespace:
Branch and git repo: https://github.com/egovernments/iFix-Dev/tree/mongodb-plainsearch/domain-services/fiscal-event-service
Build and deploy the branch “mongodb-plansearch” for these above services
Before deploying add the perisiter and the devopps file changes mentioned in step 4 and step 6
3. Once done check the count of records in mongo-db of both mgramseva and Ifix namespace and run the apis shared in point 1
After successfully running the apis check for the counts in postgres and compare the counts.
4. Persister details:
Adapter-master-data: https://github.com/misdwss/config-mgramseva/blob/UAT/egov-persister/adapter-master-data-service.yml
ifix-department-entity: https://github.com/misdwss/config-mgramseva/blob/UAT/egov-persister/department-entity-persister.yml
Ifix-master-data-service: https://github.com/misdwss/config-mgramseva/blob/master/egov-persister/ifix-master-data-persister.yml
fiscal-evenn-service: https://github.com/misdwss/config-mgramseva/blob/master/egov-persister/fiscal-events-persister.yml
5. API curls.
To Migrate Department: portforward to adapter-master-service
curl --location 'http://localhost:8080//ifix-migration-toolkit/project/v1/_migrate' \
--header 'authority: mgramseva-qa.egov.org.in' \
--header 'accept: */*' \
--header 'accept-language: en-US,en;q=0.9' \
--header 'content-type: application/json; charset=utf-8' \
--header 'cookie: _ga=GA1.1.70463156.1677473784; _ga_8H6W5DYGX0=GS1.1.1677557215.2.1.1677557273.0.0.0' \
--header 'origin: https://mgramseva-qa.egov.org.in' \
--header 'referer: https://mgramseva-qa.egov.org.in/mgramseva/home/consumercreate' \
--header 'sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"' \
--header 'sec-ch-ua-mobile: ?0' \
--header 'sec-ch-ua-platform: "macOS"' \
--header 'sec-fetch-dest: empty' \
--header 'sec-fetch-mode: cors' \
--header 'sec-fetch-site: same-origin' \
--header 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36' \
--data '{
"requestHeader": {
"apiId": "mgramseva",
"ver": 1,
"ts": "",
"action": "POST",
"did": 1,
"key": "",
"msgId": "",
"authToken": "56d054b3-206e-4a25-8c67-cbef9aa59723",
"userInfo": null
},
"tenantId":"pb"
}'
To migrate Expenditure
curl --location 'http://localhost:8080//ifix-migration-toolkit/expenditure/v1/_migrate' \
--header 'authority: mgramseva-qa.egov.org.in' \
--header 'accept: */*' \
--header 'accept-language: en-US,en;q=0.9' \
--header 'content-type: application/json; charset=utf-8' \
--header 'cookie: _ga=GA1.1.70463156.1677473784; _ga_8H6W5DYGX0=GS1.1.1677557215.2.1.1677557273.0.0.0' \
--header 'origin: https://mgramseva-qa.egov.org.in' \
--header 'referer: https://mgramseva-qa.egov.org.in/mgramseva/home/consumercreate' \
--header 'sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"' \
--header 'sec-ch-ua-mobile: ?0' \
--header 'sec-ch-ua-platform: "macOS"' \
--header 'sec-fetch-dest: empty' \
--header 'sec-fetch-mode: cors' \
--header 'sec-fetch-site: same-origin' \
--header 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36' \
--data '{
"requestHeader": {
"apiId": "mgramseva",
"ver": 1,
"ts": "",
"action": "POST",
"did": 1,
"key": "",
"msgId": "",
"authToken": "56d054b3-206e-4a25-8c67-cbef9aa59723",
"userInfo": null
},
"tenantId":"pb"
}'
To migrate department-hierarchy port forward to ifix-department-entity-service
curl --location 'http://localhost:8080//ifix-migration-toolkit/department_hierarchy/v1/_migrate' \
--header 'authority: mgramseva-qa.egov.org.in' \
--header 'accept: */*' \
--header 'accept-language: en-US,en;q=0.9' \
--header 'content-type: application/json; charset=utf-8' \
--header 'cookie: _ga=GA1.1.70463156.1677473784; _ga_8H6W5DYGX0=GS1.1.1677557215.2.1.1677557273.0.0.0' \
--header 'origin: https://mgramseva-qa.egov.org.in' \
--header 'referer: https://mgramseva-qa.egov.org.in/mgramseva/home/consumercreate' \
--header 'sec-ch-ua: "Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"' \
--header 'sec-ch-ua-mobile: ?0' \
--header 'sec-ch-ua-platform: "macOS"' \
--header 'sec-fetch-dest: empty' \
--header 'sec-fetch-mode: cors' \
--header 'sec-fetch-site: same-origin' \
--header 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36' \
--data '{
"requestHeader": {
"apiId": "mgramseva",
"ver": 1,
"ts": "",
"action": "POST",
"did": 1,
"key": "",
"msgId": "",
"authToken": "56d054b3-206e-4a25-8c67-cbef9aa59723",
"userInfo": null
},
"tenantId":"pb",
"postgresToES": true
}'To migrate Department entity
There are Intermediate calls that are made to get the details from ifix
for example when you migrate fiscal event it will call this migrate api
which in turns call this curl to get the details from fiscal-event-service
DevOps file change :
https://github.com/misdwss/iFix-DevOps/pull/841/files
https://github.com/misdwss/iFix-DevOps/pull/842/files?diff=split&w=0
https://github.com/misdwss/iFix-DevOps/pull/843/files
https://github.com/misdwss/iFix-DevOps/pull/844/files
https://github.com/misdwss/iFix-DevOps/pull/846/files