...
Click the trigger with config option
...
The DAG has 3 steps:
...
The collect-data step:
The data is collected from ES and is converted to the json file.
join_data step:
The json file is converted to csv.
The tables are joined like water and trade,water and property and trade and property.
The outcome are csv files.
upload_data step:
The csv files which are created are then uploaded to the druid by using the API.
The csv files are converted to the format as expected by the API and then uploaded to druid tables.
Postman collection for posting the data:
...