Submitting an Airflow job using the CLI

The following example demonstrates how to submit a DAG file to immediately run on CDE Airflow in Cloudera Data Engineering (CDE) using the command line interface (CLI).

Using the cde airflow submit command is a quick and efficient way of testing an Airflow job, as it spares you the task of creating and uploading resources and job definitions before running the job, and cleaning up after running the job.

This command is recommended only for Airflow DAGs that need to be run just once, because the DAG is removed from Airflow at the end of the run. To manage Airflow DAGs that need to be run more than once, or that contain schedules, use cde job run instead of this command.

To submit a DAG file to run on CDE Airflow, use the CLI command:

cde airflow submit <DAG python file> [--config <key=value>]* [--job-name <job name>]

To see the full command syntax and supported options, run cde airflow submit --help.

For example:

To submit a job with a local DAG file:

cde airflow submit my-dag.py

When the job has been submitted the CLI displays the job run ID, waits for the job to terminate, and returns an exit code of 0 for success or 1 for failure.