...
Delete the kafka connector if already exists with the kafka connection, using below command to delete connector
Code Block curl -X DELETE http://kafka-connect.mgramseva:8083/connectors/water-services-enriched-es-sink to delete connector
ws-services re-indexing
Kafka Connector Curl to be runned from playground pod
Code Block curl --location --request POST 'http://kafka-connect.mgramsevalocalhost:8083/connectors/' \ --header 'Cache-Control: no-cache' \ --header 'Content-Type: application/json' \ --header 'Postman-Token: 419e68ba-ffb9-4da9-86e1-7ad5a4c8d0b9' \ --data-raw '{ "name": "water-services-enrichedtemp-es-sink", "config": { "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector", "type.name": "general", "tasks.max": "1", "max.retries": "15", "key.ignore": "false", "retry.backoff.ms": "5000", "max.buffered.records": "25", "value.converter": "org.apache.kafka.connect.json.JsonConverter", "errors.log.enable": "true", "key.converter": "org.apache.kafka.connect.storage.StringConverter", "read.timeout.ms": "100000", "topics": "water-services-temp-enriched", "batch.size": "25", "max.in.flight.requests": "2", "schema.ignore": "true", "behavior.on.malformed.documents": "warn", "flush.timeout.ms": "3600000", "errors.deadletterqueue.topic.name": "water-services-enriched-failed", "errors.tolerance": "all", "value.converter.schemas.enable": "false", "name": "water-services-enrichedtemp-es-sink", "connection.url": "http://elasticsearch-data-v1.mgramseva:9200", "linger.ms": "1000", "transforms": "TopicNameRouter", "transforms.TopicNameRouter.type": "org.apache.kafka.connect.transforms.RegexRouter", "transforms.TopicNameRouter.regex": "water-services-temp-enriched*", "transforms.TopicNameRouter.replacement": "water-services-enriched" } }'
Invoke Legacy Index of ws-service
Code Block curl --location --request POST 'http://localhost:8055/egov-indexer/index-operations/_legacyindex' \ --header 'Content-Type: application/json' \ --data-raw '{ "RequestInfo": { "apiId": "string", "ver": "string", "ts": null, "action": "string", "did": "string", "key": "string", "msgId": "string", "authToken": "ca3256e3-5318-47b1-8a68-ffcf2228fe35", "correlationId": "e721639b-c095-40b3-86e2-acecb2cb6efb", "userInfo": { "id": 23299, "uuid": "e721639b-c095-40b3-86e2-acecb2cb6efb", "userName": "9337682030", "name": "Abhilash Seth", "type": "CITIZEN", "mobileNumber": "9337682030", "emailId": "abhilash.seth@gmail.com", "roles": [ { "id": 281, "name": "Citizen" } ] } }, "apiDetails": { "uri": "http://ws-services.mgramseva:8080/ws-services/wc/_plainsearch", "tenantIdForOpenSearch": "pb", "paginationDetails": { "offsetKey": "offset", "sizeKey": "limit", "maxPageSize": 25, "limit":25 }, "responseJsonPath": "$.WaterConnection" }, "legacyIndexTopic": "ws-connection-legacyIndex", "tenantId": "pb" }'
Delete the kafka connection after all the data has been re-indexed by follwing following below command to delete connector
Code Block |
---|
Then run curl -X DELETE http://kafka-connect.mgramseva:8083/connectors/water-services-enriched-es-sink to delete connector |
Alias water-services-enriched as water-services
KQL
Code Block |
---|
POST /_aliases
{
"actions": [
{
"add": {
"index": "water-services-enriched",
"alias": "water-services"
}
}
]
} |
...
cURL
Code Block |
---|
curl -X |
...
POST http:// |
...
elasticsearch-data- |
...
v1.mgramseva: |
...
...
9200/_aliases \ --header ' |
...
Content-Type: application/json' \
- |
...
d ' |
...
{ "actions": [ { |
...
|
...
" |
...
add": { " |
...
index": " |
...
water-services-enriched", " |
...
alias": " |
...
water-services" |
...
...
}
}
]
}' |
EChallan -Reindexing
Delete the kafka connector if already exists with the kafka connection, using below command to delete connector.Code Block curl -X DELETE http://kafka-connect.mgramseva:8083/connectors/echallan-services-enriched-es-sink
Kafka Connector Call to be runned from Playgroun podCode Block curl --location --request POST 'http://kafka-connect.mgramseva:8083/connectors/' \ --header 'Cache-Control: no-cache' \ --header 'Content-Type: application/json' \ --header 'Postman-Token: 419e68ba-ffb9-4da9-86e1-7ad5a4c8d0b9' \ --data-raw '{ "name": "echallan-services-enriched-es-sink", "config": { "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector", "type.name": "general", "tasks.max": "1", "max.retries": "15", "key.ignore": "true", "retry.backoff.ms": "5000", "max.buffered.records": "25", "value.converter": "org.apache.kafka.connect.json.JsonConverter", "errors.log.enable": "true", "key.converter": "org.apache.kafka.connect.storage.StringConverter", "read.timeout.ms": "100000", "topics": "echallan-services-enriched", "batch.size": "25", "max.in.flight.requests": "2", "schema.ignore": "true", "behavior.on.malformed.documents": "warn", "flush.timeout.ms": "3600000", "errors.deadletterqueue.topic.name": "echallan-services-enriched-failed", "errors.tolerance": "all", "value.converter.schemas.enable": "false", "name": "echallan-services-enriched-es-sink", "connection.url": "http://elasticsearch-data-v1.mgramseva:9200/", "linger.ms": "1000", "transforms": "TopicNameRouter", "transforms.TopicNameRouter.type": "org.apache.kafka.connect.transforms.RegexRouter", "transforms.TopicNameRouter.regex": "echallan-services-enriched*", "transforms.TopicNameRouter.replacement": "echallan-services-enriched" } }'
Legacy Index call from postman by port forwarding egov-indexer podCode Block curl --location --request POST 'http://localhost:8055/egov-indexer/index-operations/_legacyindex' \ --header 'Content-Type: application/json' \ --data-raw '{ "RequestInfo": { "apiId": "string", "ver": "string", "ts": null, "action": "string", "did": "string", "key": "string", "msgId": "string", "authToken": "ca3256e3-5318-47b1-8a68-ffcf2228fe35", "correlationId": "e721639b-c095-40b3-86e2-acecb2cb6efb", "userInfo": { "id": 23299, "uuid": "e721639b-c095-40b3-86e2-acecb2cb6efb", "userName": "9337682030", "name": "Abhilash Seth", "type": "CITIZEN", "mobileNumber": "9337682030", "emailId": "abhilash.seth@gmail.com", "roles": [": [ { "id": 281, "name": "Citizen" { } "id": 281, ] } }, "nameapiDetails": "Citizen"{ "uri": "http://echallan-services.mgramseva:8080/echallan-services/eChallan/v1/_plainsearch", } "tenantIdForOpenSearch": "pb", "paginationDetails": ]{ } }"offsetKey": "offset", "apiDetails": { "urisizeKey": "http://echallan-services.mgramseva:8080/echallan-services/eChallan/v1/_plainsearchlimit", "tenantIdForOpenSearchmaxPageSize": "pb"25, "paginationDetailslimit":25 { }, "offsetKeyresponseJsonPath": "offset"$.challans" }, "legacyIndexTopic": "echallan-legacyIndex", "sizeKeytenantId": "limit", pb" }'
Delete the kafka connection after all the data has been re-indexed by follwing below command to delete connector
Code Block |
---|
curl -X DELETE http://kafka-connect.mgramseva:8083/connectors/water-services-enriched-es-sink |
Alias echallan-services-enriched as echallan-services
KQL
Code Block |
---|
POST /_aliases { " |
...
actions": |
...
[ { |
...
" |
...
add": |
...
|
...
{ |
...
|
...
" |
...
index": " |
...
echallan-services-enriched", |
...
" |
...
alias": "echallan- |
...
services" } |
...
|
...
} |
...
]
} |
cURL
Code Block |
---|
...
curl -X |
...
POST http:// |
...
elasticsearch-data- |
...
v1.mgramseva: |
...
Alias echallan-services-enriched as echallan-services
...
9200/_aliases \ --header 'Content-Type: application/json' \ -d '{ "actions": [ { "add": { "index": "echallan-services-enriched", "alias": "echallan-services" } } ] }' |
Dss collection v2 re-indexing
...
Delete the kafka connector if already exists with the kafka connection, using below command to delete connector
Code Block |
---|
curl -X DELETE http://kafka-connect.mgramseva:8083/connectors/cms-case-es-sink9121 to delete connector |
Kafka Connector call to be run from playground pod
...