Overview
In this document, we will learn how to legacy index/re-index the PT Assessment re-index.
Pre-Requisites
Kubectl access to the required environment in which you want to run the re-indexing
playground pod access
Legacy index mapping/configuration done in the respective indexer-config.
Connect to the Kafka cluster and check if the connecter is available are not.
If the connector exists delete the existing connector. and then a new connector
kubectl get pods -n kafka-cluster
C:>kubectl exec -it kafka-v2-0 -n kafka-cluster bash
curl -X GET http://kafka-connect.kafka-cluster:8083/connectors/ (Checking the connector).
curl -X DELETE http://kafka-connect.kafka-cluster:8083/connectors/property-assessments-enriched-sink (delete the connector).
Create connector curl
curl -X POST 'http://kafka-connect.kafka-cluster:8083/connectors/' \
--header 'Cache-Control: no-cache' \
--header 'Content-Type: application/json' \
--header 'Postman-Token: 419e68ba-ffb9-4da9-86e1-7ad5a4c8d0b9' \
--data '{
"name": "property-assessments-enriched-sink",
"config": {
"connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
"connection.url": "http://elasticsearch-data-v1.es-cluster:9200",
"linger.ms": "1000",
"type.name": "general",
"tasks.max": "1",
"max.retries": "15",
"key.ignore": "false",
"retry.backoff.ms": "5000",
"max.buffered.records": "25",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"errors.log.enable": "true",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"read.timeout.ms": "100000",
"topics": "property-assessments-enriched",
"batch.size": "25",
"max.in.flight.requests": "2",
"schema.ignore": "true",
"behavior.on.malformed.documents": "warn",
"flush.timeout.ms": "3600000",
"errors.deadletterqueue.topic.name": "property-assessments-enriched",
"errors.tolerance": "all",
"value.converter.schemas.enable": "false",
"name": "property-assessments-enriched-sink",
"transforms": "TopicNameRouter",
"transforms.TopicNameRouter.type": "org.apache.kafka.connect.transforms.RegexRouter",
"transforms.TopicNameRouter.regex": "property-assessments-enriched*",
"transforms.TopicNameRouter.replacement": "property-assessments-enriched"
}
}'
port-forward the indexer :
eg : kubectl port-forward egov-indexer-d8f44b455-82sfc -n egov 8083:8080
Check the logs :
eg : kubectl logs egov-indexer-d8f44b455-82sfc --tail 1 -f -n egov
Re-indexing process
Starter reindexing by executing the following API from the postman.
Make sure the plansearch API is available in the specific module.
Run the API by doing the following changes.
added "pb.(ulb name )" at "tenantIdForOpenSearch": "pb.(ulb name )", and "tenantId": "pb.(ulb name )"
http://localhost:8083/egov-indexer/index-operations/_legacyindex
{
"RequestInfo": {
"userInfo": {
"id": 2396,
"userName": "EMP9",
"salutation": null,
"name": "Employ",
"gender": "MALE",
"mobileNumber": "9100000000",
"emailId": "",
"altContactNumber": null,
"pan": null,
"aadhaarNumber": null,
"permanentAddress": "Testing",
"permanentCity": "Punjab",
"permanentPinCode": "24324",
"correspondenceAddress": "pb",
"correspondenceCity": "Testing",
"correspondencePinCode": "34353",
"active": true,
"locale": null,
"type": "EMPLOYEE",
"accountLocked": false,
"accountLockedDate": 1650274886367,
"fatherOrHusbandName": "father",
"signature": "fghdfgewfg374823",
"bloodGroup": null,
"photo": null,
"identificationMark": null,
"createdBy": 102,
"lastModifiedBy": 1,
"tenantId": "pb.testing",
"roles": [
{
"code": "TL_CEMP",
"name": "TL Counter Employee",
"tenantId": "pb.testing"
},
{
"code": "NOC_CEMP",
"name": "NoC counter employee",
"tenantId": "pb.testing"
},
{
"code": "NOC_DOC_VERIFIER",
"name": "NoC Doc Verifier",
"tenantId": "pb.testing"
},
{
"code": "TL_DOC_VERIFIER",
"name": "Trade License Document verifier",
"tenantId": "pb.testing"
},
{
"code": "EMPLOYEE",
"name": "Employee",
"tenantId": "pb.testing"
},
{
"code": "PTCEMP",
"name": "PT Counter Employee",
"tenantId": "pb.testing"
},
{
"code": "UC_EMP",
"name": "Universal Collection Employee",
"tenantId": "pb.testing"
},
{
"code": "TL_FIELD_INSPECTOR",
"name": "Trade License field inspector",
"tenantId": "pb.testing"
},
{
"code": "NOC_FIELD_INSPECTOR",
"name": "NoC Field Inpector",
"tenantId": "pb.testing"
},
{
"code": "TL_APPROVER",
"name": "TL Approver",
"tenantId": "pb.testing"
},
{
"code": "PT_APPROVER",
"name": "PT Counter Approver",
"tenantId": "pb.testing"
},
{
"code": "CSR",
"name": "Customer Support Representative",
"tenantId": "pb.testing"
},
{
"code": "NOC_APPROVER",
"name": "NoC counter Approver",
"tenantId": "pb.testing"
}
],
"uuid": "8e9b3b23-78d8-4da7-8883-ab03b5b37bf2",
"createdDate": "06-09-2018 22:10:33",
"lastModifiedDate": "26-07-2022 10:55:22",
"dob": "1/7/1983",
"pwdExpiryDate": "05-12-2018 22:10:33"
}
},
"apiDetails": {
"uri": "http://property-services.egov:8080/property-services/assessment/_plainsearch",
"tenantIdForOpenSearch": "pb.abohar",
"paginationDetails": {
"offsetKey": "offset",
"sizeKey": "limit",
"maxPageSize": 100
},
"responseJsonPath": "$.Assessments"
},
"legacyIndexTopic": "pt-assessment-legacyIndex",
"tenantId": "pb.abohar"
}
Add Comment