Updating a DAG file using the CDE CLI

You can update a Directed Acyclic Graph (DAG) file using the CDE CLI for instances where the DAG needs to be overridden. For use cases where the DAG needs to be overridden, first the DAG needs to be uploaded to the resource to override the previous version, then you must update the job.

Unlike a Spark job, the Airflow job does not automatically pull in the updated resource. Airflow jobs require a forced update by calling the job update command, such that the required files are uploaded to Airflow server for processing.

Choose one of the following options in step 1:

Updating a DAG file that needs to be overridden

Run the following command in the CDE CLI:

cde resource upload --name my_pipeline_resource --local-path my_pipeline_dag.py

cde job update --name my_pipeline --dag-file my_pipeline_dag.py --mount-1-resource my_pipeline_resource

Updating a DAG in scenarios where the DAG must be set to a different one:

First upload the DAG to any resource and then the job needs to be updated. Run the following command in the CDE CLI:

cde resource upload --name my_other_pipeline_resource --local-path my_other_pipeline_dag.py

cde job update --name my_pipeline --dag-file my_other_pipeline_dag.py --mount-1-resource my_other_pipeline_resource