...
A DAG is defined in a Python script, which represents the DAGs structure (tasks and their dependencies) as code.
How to run DAG in Airflow?
Manual Trigger
1.Log onto the Punjab Prod server using the credentials:
...
3.Enter date and click on Trigger button
Format {“date” : “dd-MM-yyyy”}
...
4.The Logs can be viewed by expanding on the DAG and choosing a stage for any module and
Clicking on the Log option.
...
Configure the Airflow variables
...
Key
...
Value
...
password
...
eGov@123
...
username
...
SYSTEMSU3
...
token
...
ZWdvdi11c2VyLWNsaWVudDo=
...
tenantid
...
pg
...
usertype
...
SYSYTEM
...
totalulb_url
...
https://raw.githubusercontent.com/egovernments/punjab-mdms-data/master/data/pb/tenant/tenants.json
Configure the connections
...
ConnectionId
...
Connection Type
...
Host
...
Port
...
Schema
...
Remark
...
es_conn
...
ElasticSearch
...
elasticsearch-data-v1.es-cluster
...
9200
...
For the ES server
...
digit-auth
...
HTTP
...
...
https
...
Logs can also be viewed in the Elastic search index adaptor_logs
GET adaptor_logs/_search
the timestamp can be provided based on the day for which the logs are being searched for
...
Scheduled DAG
This DAG would trigger midnight everyday for the previous day
...
Bulk Insert for a date range
Execute this script to run the DAG for a date range for the staging NDB
https://github.com/pmidc-digit/utilities/tree/develop/egov-national-dashboard-accelerator
sh iterate_over_date.sh <start-date> <end-date>
ex: sh iterate_over_date.sh 2022-03-01 2022-03-05
date needs to be in format of YYYY-mm-dd
range is exclusive of last date, [start-date, end-date), ex: in above example, script will call dag for 1, 2, 3 and 4 march. Will not do for 5 march.