Overview
In this document, we will learn how to legacy index/re-index the PT Assessment re-index.
Pre-Requisites
Kubectl access to the required environment in which you want to run the re-indexing
playground pod access
Legacy index mapping/configuration is done in the respective indexer-config file.
Connect to the Kafka cluster and check if the connecter is available are not.
If the connector exists delete the existing connector. and then a new connector
kubectl get pods -n kafka-cluster
C:>kubectl exec -it kafka-v2-0 -n kafka-cluster bash
curl -X GET http://kafka-connect.kafka-cluster:8083/connectors/ (Checking the connector).
curl -X DELETE http://kafka-connect.kafka-cluster:8083/connectors/property-assessments-enriched-sink (delete the connector).
Create connector curl
curl -X POST 'http://kafka-connect.kafka-cluster:8083/connectors/' \
--header 'Cache-Control: no-cache' \
--header 'Content-Type: application/json' \
--header 'Postman-Token: 419e68ba-ffb9-4da9-86e1-7ad5a4c8d0b9' \
--data '{
"name": "property-assessments-enriched-sink",
"config": {
"connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
"connection.url": "http://elasticsearch-data-v1.es-cluster:9200",
"linger.ms": "1000",
"type.name": "general",
"tasks.max": "1",
"max.retries": "15",
"key.ignore": "false",
"retry.backoff.ms": "5000",
"max.buffered.records": "25",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"errors.log.enable": "true",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"read.timeout.ms": "100000",
"topics": "property-assessments-enriched",
"batch.size": "25",
"max.in.flight.requests": "2",
"schema.ignore": "true",
"behavior.on.malformed.documents": "warn",
"flush.timeout.ms": "3600000",
"errors.deadletterqueue.topic.name": "property-assessments-enriched",
"errors.tolerance": "all",
"value.converter.schemas.enable": "false",
"name": "property-assessments-enriched-sink",
"transforms": "TopicNameRouter",
"transforms.TopicNameRouter.type": "org.apache.kafka.connect.transforms.RegexRouter",
"transforms.TopicNameRouter.regex": "property-assessments-enriched*",
"transforms.TopicNameRouter.replacement": "property-assessments-enriched"
}
}'
port-forward the indexer :
eg : kubectl port-forward egov-indexer-d8f44b455-82sfc -n egov 8083:8080
Check the logs :
eg : kubectl logs egov-indexer-d8f44b455-82sfc --tail 1 -f -n egov
Re-indexing process
Starter reindexing by executing the following API from the postman.
Make sure the /_plainsearch API is available in the specific module.
Run the API by doing the following changes.
added "pb.(ulb name )" at "tenantIdForOpenSearch": "pb.(ulb name )", and "tenantId": "pb.(ulb name )"
http://localhost:8083/egov-indexer/index-operations/_legacyindex
{
"RequestInfo": {
"userInfo": {
"id": 2396,
"userName": "EMP9",
"salutation": null,
"name": "Employ",
"gender": "MALE",
"mobileNumber": "9100000000",
"emailId": "",
"altContactNumber": null,
"pan": null,
"aadhaarNumber": null,
"permanentAddress": "Testing",
"permanentCity": "Punjab",
"permanentPinCode": "24324",
"correspondenceAddress": "pb",
"correspondenceCity": "Testing",
"correspondencePinCode": "34353",
"active": true,
"locale": null,
"type": "EMPLOYEE",
"accountLocked": false,
"accountLockedDate": 1650274886367,
"fatherOrHusbandName": "father",
"signature": "fghdfgewfg374823",
"bloodGroup": null,
"photo": null,
"identificationMark": null,
"createdBy": 102,
"lastModifiedBy": 1,
"tenantId": "pb.testing",
"roles": [
{
"code": "TL_CEMP",
"name": "TL Counter Employee",
"tenantId": "pb.testing"
},
{
"code": "NOC_CEMP",
"name": "NoC counter employee",
"tenantId": "pb.testing"
},
{
"code": "NOC_DOC_VERIFIER",
"name": "NoC Doc Verifier",
"tenantId": "pb.testing"
},
{
"code": "TL_DOC_VERIFIER",
"name": "Trade License Document verifier",
"tenantId": "pb.testing"
},
{
"code": "EMPLOYEE",
"name": "Employee",
"tenantId": "pb.testing"
},
{
"code": "PTCEMP",
"name": "PT Counter Employee",
"tenantId": "pb.testing"
},
{
"code": "UC_EMP",
"name": "Universal Collection Employee",
"tenantId": "pb.testing"
},
{
"code": "TL_FIELD_INSPECTOR",
"name": "Trade License field inspector",
"tenantId": "pb.testing"
},
{
"code": "NOC_FIELD_INSPECTOR",
"name": "NoC Field Inpector",
"tenantId": "pb.testing"
},
{
"code": "TL_APPROVER",
"name": "TL Approver",
"tenantId": "pb.testing"
},
{
"code": "PT_APPROVER",
"name": "PT Counter Approver",
"tenantId": "pb.testing"
},
{
"code": "CSR",
"name": "Customer Support Representative",
"tenantId": "pb.testing"
},
{
"code": "NOC_APPROVER",
"name": "NoC counter Approver",
"tenantId": "pb.testing"
}
],
"uuid": "8e9b3b23-78d8-4da7-8883-ab03b5b37bf2",
"createdDate": "06-09-2018 22:10:33",
"lastModifiedDate": "26-07-2022 10:55:22",
"dob": "1/7/1983",
"pwdExpiryDate": "05-12-2018 22:10:33"
}
},
"apiDetails": {
"uri": "http://property-services.egov:8080/property-services/assessment/_plainsearch",
"tenantIdForOpenSearch": "pb.abohar",
"paginationDetails": {
"offsetKey": "offset",
"sizeKey": "limit",
"maxPageSize": 100
},
"responseJsonPath": "$.Assessments"
},
"legacyIndexTopic": "pt-assessment-legacyIndex",
"tenantId": "pb.abohar"
}
Add Comment