...
This DAG would trigger midnight everyday for the previous day
...
Configure the Airflow variables
...
Key
...
Value
...
password
...
eGov@123
...
username
...
SYSTEMSU3
...
token
...
ZWdvdi11c2VyLWNsaWVudDo=
...
tenantid
...
pg
...
usertype
...
SYSYTEM
...
totalulb_url
Bulk Insert for a date range
Execute this script to run the DAG for a date range for the staging NDB
...
...
Configure the connections
...
ConnectionId
...
Connection Type
...
Host
...
Port
...
Schema
...
Remark
...
es_conn
...
ElasticSearch
...
elasticsearch-data-v1.es-cluster
...
9200
...
For the ES server
...
digit-auth
...
HTTP
...
...
https
...
pmidc-digit/utilities/tree/develop/egov-national-dashboard-accelerator
sh iterate_over_date.sh <start-date> <end-date>
ex: sh iterate_over_date.sh 2022-03-01 2022-03-05
date needs to be in format of YYYY-mm-dd
range is exclusive of last date, [start-date, end-date), ex: in above example, script will call dag for 1, 2, 3 and 4 march. Will not do for 5 march.