...
This DAG would trigger midnight everyday for the previous day
...
Configure the Airflow variables
...
Key
...
Value
...
password
...
eGov@123
...
username
...
SYSTEMSU3
...
token
...
ZWdvdi11c2VyLWNsaWVudDo=
...
tenantid
...
pg
...
usertype
...
SYSYTEM
...
totalulb_url
...
https://raw.githubusercontent.com/egovernments/punjab-mdms-data/master/data/pb/tenant/tenants.json
Configure the connections
...
ConnectionId
...
Connection Type
...
Host
...
Port
...
Schema
...
Remark
...
es_conn
...
ElasticSearch
...
elasticsearch-data-v1.es-cluster
...
9200
...
For the ES server
...
digit-auth
...
HTTP
...
...
https
...
Bulk Insert for a date range
Execute this script to run the DAG for a date range for the staging NDB
sh iterate_over_date.sh <start-date> <end-date>
ex: sh iterate_over_date.sh 01-03-2022 05-03-2022
date needs to be in format of dd-mm-YYYY
range is exclusive of last date, [start-date, end-date), ex: in above example, script will call dag for 1, 2, 3 and 4 march. Will not do for 5 march.