...
Stpes to follow for re-indexing:
Delete the kafka connector if already exists with the kafka connection, using below command to delete connector
Code Block curl -X DELETE http://kafka-connect.mgramseva:8083/connectors/water-services-enriched-es-sink
ws-services re-indexing
Kafka Connector Curl to be runned from playground pod
Code Block curl --location --request POST 'http://kafka-connect.mgramsevalocalhost:8083/connectors/' \ --header 'Cache-Control: no-cache' \ --header 'Content-Type: application/json' \ --header 'Postman-Token: 419e68ba-ffb9-4da9-86e1-7ad5a4c8d0b9' \ --data-raw '{ "name": "water-services-enrichedtemp-es-sink", "config": { "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector", "type.name": "general", "tasks.max": "1", "max.retries": "15", "key.ignore": "false", "retry.backoff.ms": "5000", "max.buffered.records": "25", "value.converter": "org.apache.kafka.connect.json.JsonConverter", "errors.log.enable": "true", "key.converter": "org.apache.kafka.connect.storage.StringConverter", "read.timeout.ms": "100000", "topics": "water-services-temp-enriched", "batch.size": "25", "max.in.flight.requests": "2", "schema.ignore": "true", "behavior.on.malformed.documents": "warn", "flush.timeout.ms": "3600000", "errors.deadletterqueue.topic.name": "water-services-enriched-failed", "errors.tolerance": "all", "value.converter.schemas.enable": "false", "name": "water-services-enrichedtemp-es-sink", "connection.url": "http://elasticsearch-data-v1.mgramseva:9200", "linger.ms": "1000", "transforms": "TopicNameRouter", "transforms.TopicNameRouter.type": "org.apache.kafka.connect.transforms.RegexRouter", "transforms.TopicNameRouter.regex": "water-services-temp-enriched*", "transforms.TopicNameRouter.replacement": "water-services-enriched" } }'
Plain Search call Invoke Legacy Index of ws-service
Code Block curl --location --request POST 'http://localhost:8055/egov-indexer/index-operations/_legacyindex' \ --header 'Content-Type: application/json' \ --data-raw '{ "RequestInfo": { "apiId": "string", "ver": "string", "ts": null, "action": "string", "did": "string", "key": "string", "msgId": "string", "authToken": "ca3256e3-5318-47b1-8a68-ffcf2228fe35", "correlationId": "e721639b-c095-40b3-86e2-acecb2cb6efb", "userInfo": { "id": 23299, "uuid": "e721639b-c095-40b3-86e2-acecb2cb6efb", "userName": "9337682030", "name": "Abhilash Seth", "type": "CITIZEN", "mobileNumber": "9337682030", "emailId": "abhilash.seth@gmail.com", "roles": [ { "id": 281, "name": "Citizen" } ] } }, "apiDetails": { "uri": "http://ws-services.mgramseva:8080/ws-services/wc/_plainsearch", "tenantIdForOpenSearch": "pb", "paginationDetails": { "offsetKey": "offset", "sizeKey": "limit", "maxPageSize": 25, "limit":25 }, "responseJsonPath": "$.WaterConnection" }, "legacyIndexTopic": "ws-connection-legacyIndex", "tenantId": "pb" }'
EChallan -Reindexing
Kafka Connector Call to be runned from Playgroun pod
Delete the kafka connection after all the data has been re-indexed by following below command to delete connector
Code Block |
---|
curl - |
...
X DELETE http://kafka-connect.mgramseva:8083/connectors/ |
...
water-services- |
...
enriched-es-sink |
Alias water-services-enriched as water-services
KQL
Code Block |
---|
POST /_aliases { "actions": [ { " |
...
add": { " |
...
index": " |
...
water-services-enriched", " |
...
alias": " |
...
water-services" |
...
} |
...
|
...
} ] } |
cURL
Code Block |
---|
curl -X |
...
POST http://elasticsearch-data-v1.mgramseva:9200/_aliases \ --header 'Content-Type: application/json' \ -d '{ " |
...
actions": |
...
[ { " |
...
add": |
...
{ " |
...
index": " |
...
water-services-enriched", " |
...
alias": " |
...
water-services" } |
...
|
...
} ] }' |
EChallan -Reindexing
Delete the kafka connector if already exists with the kafka connection, using below command to delete connector.Code Block curl -X DELETE "key.converter": "org.apache.kafka.connect.storage.StringConverter", "read.timeout.ms": "100000", "topics": "echallan-services-enriched", "batch.size": "25", "max.in.flight.requests": "2http://kafka-connect.mgramseva:8083/connectors/echallan-services-enriched-es-sink
Kafka Connector Call to be runned from Playgroun pod
Legacy Index call from postmanCode Block curl --location --request POST 'http://kafka-connect.mgramseva:8083/connectors/' \ --header 'Cache-Control: no-cache' \ --header 'Content-Type: application/json' \ --header 'Postman-Token: 419e68ba-ffb9-4da9-86e1-7ad5a4c8d0b9' \ --data-raw '{ "name": "echallan-services-enriched-es-sink", "config": { "schemaconnector.ignoreclass": "trueio.confluent.connect.elasticsearch.ElasticsearchSinkConnector", "behavior.on.malformed.documentstype.name": "warngeneral", "flushtasks.timeout.msmax": "36000001", "errors.deadletterqueue.topic.namemax.retries": "echallan-services-enriched-failed15", "errorskey.toleranceignore": "alltrue", "valueretry.converterbackoff.schemas.enablems": "false5000", "namemax.buffered.records": "echallan-services-enriched-es-sink25", "connectionvalue.urlconverter": "http://elasticsearch-data-v1.mgramseva:9200/org.apache.kafka.connect.json.JsonConverter", "lingererrors.log.msenable": "1000true", "transformskey.converter": "TopicNameRouterorg.apache.kafka.connect.storage.StringConverter", "transformsread.TopicNameRoutertimeout.typems": "org.apache.kafka.connect.transforms.RegexRouter"100000", "transforms.TopicNameRouter.regextopics": "echallan-services-enriched*", "transformsbatch.TopicNameRouter.replacementsize": "echallan-services-enriched25", "max.in.flight.requests": "2", } }'
Code Block curl --location --request POST 'http://localhost:8055/egov-indexer/index-operations/_legacyindex' \ --header 'Content-Type: application/json' \ --data-raw '{ "RequestInfo": { "schema.ignore": "true", "apiId"behavior.on.malformed.documents": "stringwarn", "verflush.timeout.ms": "string3600000", "ts": nullerrors.deadletterqueue.topic.name": "echallan-services-enriched-failed", "actionerrors.tolerance": "stringall", "didvalue.converter.schemas.enable": "stringfalse", "keyname": "stringechallan-services-enriched-es-sink", "msgIdconnection.url": "stringhttp://elasticsearch-data-v1.mgramseva:9200/", "authTokenlinger.ms": "ca3256e3-5318-47b1-8a68-ffcf2228fe35",1000", "correlationIdtransforms": "e721639b-c095-40b3-86e2-acecb2cb6efbTopicNameRouter", "userInfotransforms.TopicNameRouter.type": { "org.apache.kafka.connect.transforms.RegexRouter", "id"transforms.TopicNameRouter.regex": 23299, "echallan-services-enriched*", "uuid"transforms.TopicNameRouter.replacement": "e721639bechallan-c095-40b3-86e2-acecb2cb6efb",services-enriched" } }'
Legacy Index call from postman by port forwarding egov-indexer podCode Block curl --location --request "userName": "9337682030", POST 'http://localhost:8055/egov-indexer/index-operations/_legacyindex' \ --header 'Content-Type: application/json' \ --data-raw '{ "nameRequestInfo": "Abhilash Seth", { "typeapiId": "CITIZENstring", "mobileNumberver": "9337682030string", "emailIdts": "abhilash.seth@gmail.com"null, "rolesaction": ["string", "did": "string", { "key": "string", "msgId": "string", "idauthToken": 281"ca3256e3-5318-47b1-8a68-ffcf2228fe35", "correlationId": "e721639b-c095-40b3-86e2-acecb2cb6efb", "nameuserInfo": "Citizen" { "id": 23299, } "uuid": "e721639b-c095-40b3-86e2-acecb2cb6efb", ] "userName": "9337682030", "name": "Abhilash Seth", "type": "CITIZEN", "mobileNumber": "9337682030", "emailId": "abhilash.seth@gmail.com", "roles": [ { "id": 281, "name": "Citizen" } ] } }, "apiDetails": { "uri": "http://echallan-services.mgramseva:8080/echallan-services/eChallan/v1/_plainsearch", "tenantIdForOpenSearch": "pb", "paginationDetails": { "offsetKey": "offset", "sizeKey": "limit", "maxPageSize": 25, "limit":25 }, "responseJsonPath": "$.challans" }, "legacyIndexTopic": "echallan-legacyIndex", "tenantId": "pb" }'
Delete the kafka connection after all the data has been re-indexed by follwing below command to delete connector
Code Block |
---|
curl -X DELETE http://kafka-connect.mgramseva:8083/connectors/water-services-enriched-es-sink |
Alias echallan-services-enriched as echallan-services
KQL
Code Block |
---|
POST /_aliases { "actions": [ |
...
|
...
{ |
...
|
...
" |
...
add": { " |
...
index": " |
...
echallan-services |
...
-enriched", " |
...
alias": " |
...
echallan-services" |
...
} |
...
|
...
} ] } |
cURL
Code Block |
---|
curl -X |
...
POST http://elasticsearch-data-v1.mgramseva:9200/_aliases \ --header 'Content-Type: application/json' \ -d '{ "actions": [ |
...
{ " |
...
add": |
...
{ |
...
" |
...
index": |
...
"echallan-services-enriched", " |
...
alias": " |
...
echallan-services" |
...
} |
...
|
...
} |
...
] }' |
...
Dss collection v2 re-indexing
Delete the kafka connector if already exists with the kafka connection, using below command to delete connector
Code Block |
---|
curl -X DELETE http://kafka-connect.mgramseva:8083/connectors/cms-case-es-sink9121 |
Kafka Connector call to be run from playground pod
...
Code Block |
---|
curl --location --request POST 'http://localhost:8055/egov-indexer/index-operations/_legacyindex' \ --header 'Content-Type: application/json' \ --data-raw '{ "RequestInfo": { "apiId": "string", "ver": "string", "ts": null, "action": "string", "did": "string", "key": "string", "msgId": "string", "authToken": "b843ef27-1ac6-49b8-ab71-cd0c22f4e50e", "correlationId": "e721639b-c095-40b3-86e2-acecb2cb6efb", "userInfo": { "id": 23299, "uuid": "e721639b-c095-40b3-86e2-acecb2cb6efb", "userName": "9337682030", "name": "Abhilash Seth", "type": "EMPLOYEE", "mobileNumber": "9337682030", "emailId": "abhilash.seth@gmail.com", "roles": [ { "id": 281, "name": "Employee" } ] } }, "apiDetails": { "uri": "http://collection-services.mgramseva:8080/collection-services/payments/_plainsearch", "tenantIdForOpenSearch": "pb", "paginationDetails": { "offsetKey": "offset", "sizeKey": "limit", "maxPageSize": 50, "startingOffset": 1500 }, "responseJsonPath": "$.Payments" }, "legacyIndexTopic": "egov-payment-legacy-index", "tenantId": "pb" }' |
...