Page Comparison

...

Delete the kafka connector if already exists with the kafka connection, using below command to delete connector

Code Block
Then run curl -X DELETE http://kafka-connect.mgramseva:8083/connectors/water-services-enriched-es-sink to delete connector

ws-services re-indexing

Kafka Connector Curl to be runned from playground pod

Code Block

curl --location --request POST 'http://kafka-connect.mgramsevalocalhost:8083/connectors/' \
--header 'Cache-Control: no-cache' \
--header 'Content-Type: application/json' \
--header 'Postman-Token: 419e68ba-ffb9-4da9-86e1-7ad5a4c8d0b9' \
--data-raw '{
    "name": "water-services-enrichedtemp-es-sink",
    "config": {
        "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
        "type.name": "general",
        "tasks.max": "1",
        "max.retries": "15",
        "key.ignore": "false",
        "retry.backoff.ms": "5000",
        "max.buffered.records": "25",
        "value.converter": "org.apache.kafka.connect.json.JsonConverter",
        "errors.log.enable": "true",
        "key.converter": "org.apache.kafka.connect.storage.StringConverter",
        "read.timeout.ms": "100000",
        "topics": "water-services-temp-enriched",
        "batch.size": "25",
        "max.in.flight.requests": "2",
        "schema.ignore": "true",
        "behavior.on.malformed.documents": "warn",
        "flush.timeout.ms": "3600000",
        "errors.deadletterqueue.topic.name": "water-services-enriched-failed",
        "errors.tolerance": "all",
        "value.converter.schemas.enable": "false",
        "name": "water-services-enrichedtemp-es-sink",
        "connection.url": "http://elasticsearch-data-v1.mgramseva:9200",
        "linger.ms": "1000",
        "transforms": "TopicNameRouter",
        "transforms.TopicNameRouter.type": "org.apache.kafka.connect.transforms.RegexRouter",
        "transforms.TopicNameRouter.regex": "water-services-temp-enriched*",
        "transforms.TopicNameRouter.replacement": "water-services-enriched"
    }
}'

Invoke Legacy Index of ws-service

Code Block

curl --location --request POST 'http://localhost:8055/egov-indexer/index-operations/_legacyindex' \
--header 'Content-Type: application/json' \
--data-raw '{
    "RequestInfo": {
        "apiId": "string",
        "ver": "string",
        "ts": null,
        "action": "string",
        "did": "string",
        "key": "string",
        "msgId": "string",
        "authToken": "ca3256e3-5318-47b1-8a68-ffcf2228fe35",
        "correlationId": "e721639b-c095-40b3-86e2-acecb2cb6efb",
        "userInfo": {
            "id": 23299,
            "uuid": "e721639b-c095-40b3-86e2-acecb2cb6efb",
            "userName": "9337682030",
            "name": "Abhilash Seth",
            "type": "CITIZEN",
            "mobileNumber": "9337682030",
            "emailId": "abhilash.seth@gmail.com",
            "roles": [
                {
                    "id": 281,
                    "name": "Citizen"
                }
            ]
        }
    },
    "apiDetails": {
        "uri": "http://ws-services.mgramseva:8080/ws-services/wc/_plainsearch",
        "tenantIdForOpenSearch": "pb",
        "paginationDetails": {
            "offsetKey": "offset",
            "sizeKey": "limit",
            "maxPageSize": 25,
            "limit":25
        },
        "responseJsonPath": "$.WaterConnection"
    },
    "legacyIndexTopic": "ws-connection-legacyIndex",
    "tenantId": "pb"  
}'

Delete the kafka connection after all the data has been re-indexed by follwing following below command to delete connector

Code Block
Then run curl -X DELETE http://kafka-connect.mgramseva:8083/connectors/water-services-enriched-es-sink to delete connector

...

Code Block
Then run curl -X DELETE http://kafka-connect.mgramseva:8083/connectors/echallan-services-enriched-es-sink to delete connector

...

Alias water-services-enriched as water-services

KQL

Code Block
POST /_aliases { "actions": [ { "add": { "index": "water-services-enriched", "alias": "water-services" } } ] }

cURL

Code Block
curl -

...

X POST

...

http://

...

elasticsearch-data-

...

v1.mgramseva:

...

9200/_aliases \
--header '

...

Content-

...

Type:

...

application/json' \
-

...

d '

...

{
  "actions": [
    {
      "add": {
        "

...

index": "

...

water-services-enriched

...

",
        "

...

alias":

...

"water-services"
      }

...

 }
  ]
}'

EChallan -Reindexing

Delete the kafka connector if already exists with the kafka connection, using below command to delete connector.

Code Block
curl -X DELETE http://kafka-connect.mgramseva:8083/connectors/echallan-services-enriched-es-sink

Kafka Connector Call to be runned from Playgroun pod

Code Block

curl --location --request POST 'http://kafka-connect.mgramseva:8083/connectors/' \
--header 'Cache-Control: no-cache' \
--header 'Content-Type: application/json' \
--header 'Postman-Token: 419e68ba-ffb9-4da9-86e1-7ad5a4c8d0b9' \
--data-raw '{
    "name": "echallan-services-enriched-es-sink",
    "config": {
        "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
        "type.name": "general",
        "tasks.max": "1",
        "max.retries": "15",
        "key.ignore": "true",
        "retry.backoff.ms": "5000",
        "max.buffered.records": "25",
        "value.converter": "org.apache.kafka.connect.json.JsonConverter",
        "errors.log.enable": "true",
        "key.converter": "org.apache.kafka.connect.storage.StringConverter",
        "read.timeout.ms": "100000",
        "topics": "echallan-services-enriched",
        "batch.size": "25",
        "type.namemax.in.flight.requests": "general2",
        "tasksschema.maxignore": "1true",
        "max.retriesbehavior.on.malformed.documents": "15warn",
        "keyflush.timeout.ignorems": "true3600000",
        "retryerrors.deadletterqueue.backofftopic.msname": "5000echallan-services-enriched-failed",
        "maxerrors.buffered.recordstolerance": "25all",
        "value.converter.schemas.enable": "org.apache.kafka.connect.json.JsonConverterfalse",
        "errors.log.enablename": "trueechallan-services-enriched-es-sink",
        "keyconnection.converterurl": "org.apache.kafka.connect.storage.StringConverterhttp://elasticsearch-data-v1.mgramseva:9200/",
        "readlinger.timeout.ms": "1000001000",
        "topicstransforms": "echallan-services-enrichedTopicNameRouter",
        "batchtransforms.TopicNameRouter.sizetype": "25org.apache.kafka.connect.transforms.RegexRouter",
        "maxtransforms.inTopicNameRouter.flight.requestsregex": "2echallan-services-enriched*",
        "schematransforms.TopicNameRouter.ignorereplacement": "true",
        "behavior.on.malformed.documents": "warn",echallan-services-enriched"
    }
}'

Legacy Index call from postman by port forwarding egov-indexer pod

Code Block

curl --location --request  "flush.timeout.ms": "3600000",
   POST 'http://localhost:8055/egov-indexer/index-operations/_legacyindex' \
--header 'Content-Type: application/json' \
--data-raw '{
    "errors.deadletterqueue.topic.nameRequestInfo": "echallan-services-enriched-failed",{
        "errors.toleranceapiId": "allstring",
        "value.converter.schemas.enablever": "falsestring",
        "namets": "echallan-services-enriched-es-sink"null,
        "connection.urlaction": "http://elasticsearch-data-v1.mgramseva:9200/string",
        "linger.msdid": "1000string",
        "transformskey": "TopicNameRouterstring",
        "transforms.TopicNameRouter.typemsgId": "org.apache.kafka.connect.transforms.RegexRouterstring",
        "transforms.TopicNameRouter.regexauthToken": "echallan-services-enriched*ca3256e3-5318-47b1-8a68-ffcf2228fe35",
        "transforms.TopicNameRouter.replacementcorrelationId": "echallan-services-enriched"
    }
}'

Legacy Index call from postman

Code Block

curl --location --request POST 'http://localhost:8055/egov-indexer/index-operations/_legacyindex' \
--header 'Content-Type: application/json' \
--data-raw '{e721639b-c095-40b3-86e2-acecb2cb6efb",
        "userInfo": {
            "RequestInfoid": { 23299,
            "apiIduuid": "stringe721639b-c095-40b3-86e2-acecb2cb6efb",
            "veruserName": "string9337682030",

       "ts": null,         "actionname": "stringAbhilash Seth",
            "didtype": "stringCITIZEN",
            "keymobileNumber": "string9337682030",
            "msgIdemailId": "stringabhilash.seth@gmail.com",
            "authTokenroles": "ca3256e3-5318-47b1-8a68-ffcf2228fe35", [
             "correlationId": "e721639b-c095-40b3-86e2-acecb2cb6efb",  {
      "userInfo": {             "id": 23299281,
            "uuid": "e721639b-c095-40b3-86e2-acecb2cb6efb",        "name": "Citizen"
     "userName": "9337682030",          }
  "name": "Abhilash Seth",        ]
    "type": "CITIZEN",   }
    },
    "mobileNumberapiDetails": "9337682030",
   {
        "emailIduri": "abhilash.seth@gmail.com",
  http://echallan-services.mgramseva:8080/echallan-services/eChallan/v1/_plainsearch",
         "rolestenantIdForOpenSearch": ["pb",
               "paginationDetails": {
            "offsetKey": "offset",
      "id": 281,         "sizeKey": "limit",
            "namemaxPageSize": "Citizen"25,
            "limit":25
     }   },
         ]"responseJsonPath": "$.challans"
    },
   } "legacyIndexTopic": "echallan-legacyIndex",
  },  "tenantId": "pb"
 "apiDetails": {
        "uri": "}'

Delete the kafka connection after all the data has been re-indexed by follwing below command to delete connector

Code Block
curl -X DELETE http://

...

kafka-

...

connect.mgramseva:

...

8083/connectors/water-services-enriched-es-sink

Alias echallan-services

...

-enriched as echallan-services

KQL

Code Block
POST /_aliases { "

...

actions":

...

 [
    {
      "

...

add": {

...

index": "

...

echallan-services-enriched",

...

alias": "

...

echallan-services"

...


      }
    }

...

]
}

cURL

Code Block
curl -X POST http://elasticsearch-data-v1.mgramseva:9200/_aliases \ --header 'Content-Type: application/json' \ -d '{ "

...

actions":

...

add":

...

index": "echallan-

...

services-enriched",

...

Delete the kafka connection after all the data has been re-indexed by follwing below command

...

        "alias": "echallan-services"
      }
    }
  ]
}'

Dss collection v2 re-indexing

...

Delete the kafka connector if already exists with the kafka connection, using below command to delete connector

Code Block
Then run curl -X DELETE http://kafka-connect.mgramseva:8083/connectors/cms-case-es-sink9121 to delete connector

Kafka Connector call to be run from playground pod

...

Code Block

curl --location --request POST 'http://localhost:8055/egov-indexer/index-operations/_legacyindex' \
--header 'Content-Type: application/json' \
--data-raw '{
    "RequestInfo": {
        "apiId": "string",
        "ver": "string",
        "ts": null,
        "action": "string",
        "did": "string",
        "key": "string",
        "msgId": "string",
        "authToken": "b843ef27-1ac6-49b8-ab71-cd0c22f4e50e",
        "correlationId": "e721639b-c095-40b3-86e2-acecb2cb6efb",
        "userInfo": {
            "id": 23299,
            "uuid": "e721639b-c095-40b3-86e2-acecb2cb6efb",
            "userName": "9337682030",
            "name": "Abhilash Seth",
            "type": "EMPLOYEE",
            "mobileNumber": "9337682030",
            "emailId": "abhilash.seth@gmail.com",
            "roles": [
                {
                    "id": 281,
                    "name": "Employee"
                }
            ]
        }
    },
    "apiDetails": {
        "uri": "http://collection-services.mgramseva:8080/collection-services/payments/_plainsearch",
          "tenantIdForOpenSearch": "pb",
        "paginationDetails": {
            "offsetKey": "offset",
            "sizeKey": "limit",
            "maxPageSize": 50,
            "startingOffset": 1500
        },
        "responseJsonPath": "$.Payments"
    },
    "legacyIndexTopic": "egov-payment-legacy-index",
    "tenantId": "pb"
}'

...

Versions Compared

Old Version 3

New Version Current

Key