Versions Compared

Old Version 2

changes.mady.by.user Pintu

Saved on

compared with

New Version 3

changes.mady.by.user Pintu

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Fiscal Event Post Processor is a streaming pipeline for validated fiscal event data. Apache Kafka has been used to stream the validated fiscal event data, process it for dereference, unbundle, flatten And finally push these details to Mongo and Druid data store.

Version

Current version : 12.10.0

Prerequisites

Before you proceed with the configuration, make sure the following pre-requisites are met

  1. Java 8

  2. Apache Kafka and Kafka-Connect server should be up and running

  3. Druid DB & MongoDB should be up and running

  4. Below dependent services are required:
    iFix Master data service
    iFix Fiscal Event service

...

  1. Fiscal event validated data will get dereferenced. For dereferencing, pass the service ids like project id, COA id, Tenant id etc. to corresponding services - Master service & Department Entity service And get the corresponding object(s). Once the fiscal event data is dereferenced, push/send the same data to Mongo Sink and dereference Topic.

  2. Mongo connector will pick up the data from the “fiscal-event-mongodb-sink“ topic and push it to Mongo Datastore.

  3. Unbundle consumers pick up the dereferenced fiscal event data from dereferencing topic. Dereference fiscal event data gets unbundled and then flattened. Once the flattening is complete, push/send the same data to Druid Sink topic.

  4. Flattened fiscal event data is pushed to Druid DB from a topic named: fiscal-event-druid-sink.

...

  1. Open the Druid console

  2. Go to the Load Data section

  3. Select Other

  4. Click on Submit Supervisor

  5. Copy...Paste the JSON from the the druid-ingestion-config.json file file in the available text box.

  6. Verify the Kafka topic name and the Kafka bootstrap server address before submitting the config

  7. Submit the config and the data ingestion should start into the fiscal-event data source

Interaction Diagram

Image RemovedImage Added

Environment

Note: Kafka topic needs to be configured with respect to the environment

Key

Value

Description

Remarks

fiscal-event-kafka-push-topic

fiscal-event-request-validated

Fiscal event post processor will consume data from this topic

Kafka topic should be same as configured in Fiscal event service.

fiscal-event-kafka-dereferenced-topic

fiscal-event-request-dereferenced

Dereferenced fiscal event data will be pushed to this topic

NA

fiscal-event-kafka-flattened-topic

fiscal-event-line-item-flattened

NA

NAfiscal-event-processor-kafka-mongodb-topic

fiscal-event-mongodb-sink

Dereferenced fiscal event data will be pushed to this topic and will consumed Kafka connect to persist in Mongo Data Store.

Mongo Kafka connects topic should be same.

fiscal-event-processor-kafka-druid-topic

fiscal-event-druid-sink

Flattened Fiscal Event data will be pushed to this topic.

While druid ingest of fiscal event , make sure it has the same topic as mentioned here.

...

References and Notes

Title

Link

Swagger Yaml

https://github.com/egovernmentsmisdwss/iFix-Dev/blob/developifix-2.0-alpha/domain-services/fiscal-event-post-processor/fiscal-event-post-processor-12.0.0.yaml