Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 5 Next »

In Airflow, a DAG –Directed Acyclic Graph – is a collection of all the tasks you want to run, organized in a way that reflects their relationships and dependencies.

A DAG is defined in a Python script, which represents the DAGs structure (tasks and their dependencies) as code.

How to run DAG in Airflow?

Manual Trigger

1.Log onto the Punjab Prod server using the credentials:

URL:https://airflow.mseva-qa.lgpunjab.gov.in/login/?next=http%3A%2F%2Fairflow.mseva-qa.lgpunjab.gov.in%2Fhome

    username : admin

    password :  admin

 

 

2.Trigger the DAG by clicking on the “Trigger DAG with Config” option.

3.Enter date and click on Trigger button

Format {“date” : “dd-MM-yyyy”}

4.The Logs can be viewed by expanding on the DAG and choosing a stage for any module and 

Clicking on the Log option.

Logs can also be viewed in the Elastic search index adaptor_logs

GET adaptor_logs/_search

the timestamp can be provided based on the day for which the logs are being searched for

Scheduled DAG

This DAG would trigger midnight everyday for the previous day

Bulk Insert for a date range

Execute this script to run the DAG for a date range for the staging NDB

sh iterate_over_date.sh <start-date> <end-date>
ex: sh iterate_over_date.sh 01-03-2022 05-03-2022

  • date needs to be in format of dd-mm-YYYY

  • range is exclusive of last date, [start-date, end-date), ex: in above example, script will call dag for 1, 2, 3 and 4 march. Will not do for 5 march.

https://github.com/pmidc-digit/utilities/blob/develop/egov-national-dashboard-accelerator/Bulk_insert.sh

  • No labels