iFIX: Migration from Druid/Metabase to ElasticSearch/DSS
Overview
Due to certain limitations of Druid and Metabase, we decided to move the iFIX DWSS dashboard to ElasticSearch/DSS. As a consequence of this, we had to add a couple of new services like -
iFIX Migration Toolkit - This utility service helps in moving fiscal data from MongoDB to ElasticSearch.
iFIX ES Pipeline - This pipeline service listens onto fiscal data(both existing and new ones), enriches them, performs certain pre-computations so as to make writing dashboard queries efficient and simple to write.
Indexer Service - Indexer service was deployed to index fiscal records onto ElasticSearch.
Persister Service - Persister service was deployed to persist migration progress entries which we are maintaining so as to avoid duplication of records. Persister in turn stores records in PostgreSQL database.
Dashboard Analytics Service - Dashboard analytics service was deployed to create charts out of fiscal data which was moved to ElasticSearch.
DIGIT-UI - A customized version of DIGIT-UI was deployed in order to look at the new dashboard.
Other than that, we had to add new features to our existing services as follows -
Fiscal Event Service - A plain search API was added in order to help migration toolkit utility service to fetch existing records from MongoDB in a paginated manner and to maintain migration progress entries. Other than that, we made a little change to fiscal event service’s mapping method where it maps nodal data into FiscalEvent POJOs.
Now, the overall flow for indexing of new fiscal events looks like -
The overall flow for migration looks like -
