Property Tax Assessment Re-indexing/Legacy

Overview

In this document, we will learn how to legacy index/re-index the PT Assessment re-index.

Pre-Requisites

Kubectl access to the required environment in which you want to run the re-indexing
playground pod access
Legacy index mapping/configuration is done in the respective indexer-config file.

Connect to the Kafka cluster and check if the connecter is available or not.
If the connector exists delete the existing connector. and then a new connector

kubectl get pods -n kafka-cluster

C:>kubectl exec -it kafka-v2-0 -n kafka-cluster bash

curl -X GET http://kafka-connect.kafka-cluster:8083/connectors/ (Checking the connector).

curl -X DELETE http://kafka-connect.kafka-cluster:8083/connectors/property-assessments-enriched-sink (delete the connector).

Create connector curl

 curl -X POST 'http://kafka-connect.kafka-cluster:8083/connectors/' \
 --header 'Cache-Control: no-cache' \
 --header 'Content-Type: application/json' \
 --header 'Postman-Token: 419e68ba-ffb9-4da9-86e1-7ad5a4c8d0b9' \
 --data '{
     "name": "property-assessments-enriched-sink",
     "config": {
         "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
         "connection.url": "http://elasticsearch-data-v1.es-cluster:9200",
         "linger.ms": "1000",
         "type.name": "general",
         "tasks.max": "1",
         "max.retries": "15",
         "key.ignore": "false",
         "retry.backoff.ms": "5000",
         "max.buffered.records": "25",
         "value.converter": "org.apache.kafka.connect.json.JsonConverter",
         "errors.log.enable": "true",
         "key.converter": "org.apache.kafka.connect.storage.StringConverter",
         "read.timeout.ms": "100000",
         "topics": "property-assessments-enriched",
         "batch.size": "25",
         "max.in.flight.requests": "2",
         "schema.ignore": "true",
         "behavior.on.malformed.documents": "warn",
         "flush.timeout.ms": "3600000",
         "errors.deadletterqueue.topic.name": "property-assessments-enriched",
         "errors.tolerance": "all",
         "value.converter.schemas.enable": "false",
         "name": "property-assessments-enriched-sink",

         "transforms": "TopicNameRouter",
         "transforms.TopicNameRouter.type": "org.apache.kafka.connect.transforms.RegexRouter",
         "transforms.TopicNameRouter.regex": "property-assessments-enriched*",
         "transforms.TopicNameRouter.replacement": "property-assessments-enriched"
     }
 }'

port-forward the indexer :

eg : kubectl port-forward egov-indexer-d8f44b455-82sfc -n egov 8083:8080

Check the logs :

eg : kubectl logs egov-indexer-d8f44b455-82sfc --tail 1 -f -n egov

Re-indexing process

Starter reindexing by executing the following API from the postman.
Make sure the /_plainsearch API is available in the specific module.
Run the API by doing the following changes.
added "pb.(ulb name )" at "tenantIdForOpenSearch": "pb.(ulb name )", and "tenantId": "pb.(ulb name )"

http://localhost:8083/egov-indexer/index-operations/_legacyindex

{
    "RequestInfo": {
        "userInfo": {
            "id": 2396,
            "userName": "EMP9",
            "salutation": null,
            "name": "Employ",
            "gender": "MALE",
            "mobileNumber": "9100000000",
            "emailId": "",
            "altContactNumber": null,
            "pan": null,
            "aadhaarNumber": null,
            "permanentAddress": "Testing",
            "permanentCity": "Punjab",
            "permanentPinCode": "24324",
            "correspondenceAddress": "pb",
            "correspondenceCity": "Testing",
            "correspondencePinCode": "34353",
            "active": true,
            "locale": null,
            "type": "EMPLOYEE",
            "accountLocked": false,
            "accountLockedDate": 1650274886367,
            "fatherOrHusbandName": "father",
            "signature": "fghdfgewfg374823",
            "bloodGroup": null,
            "photo": null,
            "identificationMark": null,
            "createdBy": 102,
            "lastModifiedBy": 1,
            "tenantId": "pb.testing",
            "roles": [
                {
                    "code": "TL_CEMP",
                    "name": "TL Counter Employee",
                    "tenantId": "pb.testing"
                },
                {
                    "code": "NOC_CEMP",
                    "name": "NoC counter employee",
                    "tenantId": "pb.testing"
                },
                {
                    "code": "NOC_DOC_VERIFIER",
                    "name": "NoC Doc Verifier",
                    "tenantId": "pb.testing"
                },
                {
                    "code": "TL_DOC_VERIFIER",
                    "name": "Trade License Document verifier",
                    "tenantId": "pb.testing"
                },
                {
                    "code": "EMPLOYEE",
                    "name": "Employee",
                    "tenantId": "pb.testing"
                },
                {
                    "code": "PTCEMP",
                    "name": "PT Counter Employee",
                    "tenantId": "pb.testing"
                },
                {
                    "code": "UC_EMP",
                    "name": "Universal Collection Employee",
                    "tenantId": "pb.testing"
                },
                {
                    "code": "TL_FIELD_INSPECTOR",
                    "name": "Trade License field inspector",
                    "tenantId": "pb.testing"
                },
                {
                    "code": "NOC_FIELD_INSPECTOR",
                    "name": "NoC Field Inpector",
                    "tenantId": "pb.testing"
                },
                {
                    "code": "TL_APPROVER",
                    "name": "TL Approver",
                    "tenantId": "pb.testing"
                },
                {
                    "code": "PT_APPROVER",
                    "name": "PT Counter Approver",
                    "tenantId": "pb.testing"
                },
                {
                    "code": "CSR",
                    "name": "Customer Support Representative",
                    "tenantId": "pb.testing"
                },
                {
                    "code": "NOC_APPROVER",
                    "name": "NoC counter Approver",
                    "tenantId": "pb.testing"
                }
            ],
            "uuid": "8e9b3b23-78d8-4da7-8883-ab03b5b37bf2",
            "createdDate": "06-09-2018 22:10:33",
            "lastModifiedDate": "26-07-2022 10:55:22",
            "dob": "1/7/1983",
            "pwdExpiryDate": "05-12-2018 22:10:33"
        }
    },
    "apiDetails": {
        "uri": "http://property-services.egov:8080/property-services/assessment/_plainsearch",
        "tenantIdForOpenSearch": "pb.abohar",
        "paginationDetails": {
            "offsetKey": "offset",
            "sizeKey": "limit",
            "maxPageSize": 100
        },
        "responseJsonPath": "$.Assessments"
    },
    "legacyIndexTopic": "pt-assessment-legacyIndex",
    "tenantId": "pb.abohar"
}