...
Clicking on the Log option.
...
Logs can also be viewed in the Elastic search index adaptor_logs
GET adaptor_logs/_search
the timestamp can be provided based on the day for which the logs are being searched for
...
Scheduled DAG
This DAG would trigger midnight everyday for the previous day
...
Configure the Airflow variables
...
Key
...
Value
...
password
...
eGov@123
...
username
...
SYSTEMSU3
...
token
...
ZWdvdi11c2VyLWNsaWVudDo=
...
tenantid
...
pg
...
usertype
...
SYSYTEM
...
totalulb_url
...
https://raw.githubusercontent.com/egovernments/punjab-mdms-data/master/data/pb/tenant/tenants.json
Configure the connections
...
ConnectionId
...
Connection Type
...
Host
...
Port
...
Schema
...
Remark
...
es_conn
...
ElasticSearch
...
elasticsearch-data-v1.es-cluster
...
9200
...
For the ES server
...
digit-auth
...
HTTP
...
...
https
...
Bulk Insert for a date range
Execute this script to run the DAG for a date range for the staging NDB
https://github.com/pmidc-digit/utilities/tree/develop/egov-national-dashboard-accelerator
sh iterate_over_date.sh <start-date> <end-date>
ex: sh iterate_over_date.sh 2022-03-01 2022-03-05
date needs to be in format of YYYY-mm-dd
range is exclusive of last date, [start-date, end-date), ex: in above example, script will call dag for 1, 2, 3 and 4 march. Will not do for 5 march.