Druid Issues

Owned by Rushang Dhanesha
May 30, 2022
1 min read

Whenever there is a surge in the incoming records, Druid requires extra resources to ingest all those records.

So while migrating fiscal events from MongoDB to Druid, we need to scale up the Druid cluster. Two components of the Druid cluster are responsible for ingestion:

  1. Middle Manager: We should provide extra memory to the middle manager. Increasing the number of pods just for the purpose of migration is not advisable.

  2. Overlord: Overlord can be scaled both horizontally(increasing number of pods) and vertically(increasing memory).

Below are the examples of the necessary changes required in the environment file of the DevOps repo:

  1. Middle Manager: https://github.com/misdwss/iFix-DevOps/commit/5e6056bd4b3c78f536d4bf1d0f7b173b837d96eb

  2. Overlord: https://github.com/misdwss/iFix-DevOps/commit/22717cca52c60ac36c1d2e07a28d40f7a95c445e

You should revery back to the original resources after the migration is successfully complete.

Important Note:

If there is lack of memory in the kuberenetes cluster, then please add node to the cluster before allocating any extra memory.