mGramseva- Services Re-Indexing

Owned by lakshmi.sravani
Last updated: Aug 30, 2023 by snehal.gothe (Unlicensed)
10 min read

Overview:

We are using re-indexing to get all the data to respective indexer. We have 2 steps for this. First as to run the connector from playground, which is followed by legacyindexer service call from indexer service, which internally call the respective plain search service to get the data and to send to respective indexer.


Prerequisites:

  • Access to kubectl of the environment targetted

  • Postman scripts

  • Plain search apis in the respective services

 

We have mainly 3 indexes in mGramseva for Re-indexing.

  • Water-services

  • Echallan-services

  • dss-collection_v2


Stpes to follow for re-indexing:

  • Delete the kafka connector if already exists with the kafka connection, using below command to delete connector

    curl -X DELETE http://kafka-connect.mgramseva:8083/connectors/water-services-enriched-es-sink

  • ws-services re-indexing

    Kafka Connector Curl to be runned from playground pod

    curl --location --request POST 'http://localhost:8083/connectors/' \ --header 'Cache-Control: no-cache' \ --header 'Content-Type: application/json' \ --header 'Postman-Token: 419e68ba-ffb9-4da9-86e1-7ad5a4c8d0b9' \ --data '{ "name": "water-services-temp-es-sink", "config": { "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector", "type.name": "general", "tasks.max": "1", "max.retries": "15", "key.ignore": "false", "retry.backoff.ms": "5000", "max.buffered.records": "25", "value.converter": "org.apache.kafka.connect.json.JsonConverter", "errors.log.enable": "true", "key.converter": "org.apache.kafka.connect.storage.StringConverter", "read.timeout.ms": "100000", "topics": "water-services-temp-enriched", "batch.size": "25", "max.in.flight.requests": "2", "schema.ignore": "true", "behavior.on.malformed.documents": "warn", "flush.timeout.ms": "3600000", "errors.deadletterqueue.topic.name": "water-services-enriched-failed", "errors.tolerance": "all", "value.converter.schemas.enable": "false", "name": "water-services-temp-es-sink", "connection.url": "http://elasticsearch-data-v1.mgramseva:9200", "linger.ms": "1000", "transforms": "TopicNameRouter", "transforms.TopicNameRouter.type": "org.apache.kafka.connect.transforms.RegexRouter", "transforms.TopicNameRouter.regex": "water-services-temp-enriched*", "transforms.TopicNameRouter.replacement": "water-services" } }'

     

     

    Invoke Legacy Index of ws-service

     

    curl --location --request POST 'http://localhost:8055/egov-indexer/index-operations/_legacyindex' \ --header 'Content-Type: application/json' \ --data-raw '{ "RequestInfo": { "apiId": "string", "ver": "string", "ts": null, "action": "string", "did": "string", "key": "string", "msgId": "string", "authToken": "ca3256e3-5318-47b1-8a68-ffcf2228fe35", "correlationId": "e721639b-c095-40b3-86e2-acecb2cb6efb", "userInfo": { "id": 23299, "uuid": "e721639b-c095-40b3-86e2-acecb2cb6efb", "userName": "9337682030", "name": "Abhilash Seth", "type": "CITIZEN", "mobileNumber": "9337682030", "emailId": "abhilash.seth@gmail.com", "roles": [ { "id": 281, "name": "Citizen" } ] } }, "apiDetails": { "uri": "http://ws-services.mgramseva:8080/ws-services/wc/_plainsearch", "tenantIdForOpenSearch": "pb", "paginationDetails": { "offsetKey": "offset", "sizeKey": "limit", "maxPageSize": 25, "limit":25 }, "responseJsonPath": "$.WaterConnection" }, "legacyIndexTopic": "ws-connection-legacyIndex", "tenantId": "pb" }'

     

Delete the kafka connection after all the data has been re-indexed by following below command to delete connector

 

 

Alias water-services-enriched as water-services

KQL

 

cURL

 

 

 

  • EChallan -Reindexing

    Delete the kafka connector if already exists with the kafka connection, using below command to delete connector.



    Kafka Connector Call to be runned from Playgroun pod

     


    Legacy Index call from postman by port forwarding egov-indexer pod

Delete the kafka connection after all the data has been re-indexed by follwing below command to delete connector

 

 

Alias echallan-services-enriched as echallan-services

KQL

cURL

 

 

  • Dss collection v2 re-indexing

 

Delete the kafka connector if already exists with the kafka connection, using below command to delete connector



Kafka Connector call to be run from playground pod

 

 

 

 

payment re-indexing run from postman call