In Airflow, a DAG
–Directed Acyclic Graph – is a collection of all the tasks you want to run, organized in a way that reflects their relationships and dependencies.
A DAG is defined in a Python script, which represents the DAGs structure (tasks and their dependencies) as code.
How to run DAG in Airflow?
Manual Trigger
1.Log onto the Punjab Prod server using the credentials:
username : admin
password : admin
2.Trigger the DAG by clicking on the “Trigger DAG with Config” option.
3.Enter date and click on Trigger button
Format {“date” : “dd-MM-yyyy”}
4.The Logs can be viewed by expanding on the DAG and choosing a stage for any module and
Clicking on the Log option.
Scheduled DAG
This DAG would trigger midnight everyday for the previous day
Configure the Airflow variables
Key | Value |
---|---|
password | eGov@123 |
username | SYSTEMSU3 |
token | ZWdvdi11c2VyLWNsaWVudDo= |
tenantid | pg |
usertype | SYSYTEM |
totalulb_url | https://raw.githubusercontent.com/egovernments/punjab-mdms-data/master/data/pb/tenant/tenants.json |
Configure the connections
ConnectionId | Connection Type | Host | Port | Schema | Remark |
es_conn | ElasticSearch | elasticsearch-data-v1.es-cluster | 9200 | For the ES server | |
digit-auth | HTTP | https | For the auth api conenction |
Add Comment