Submitting an Airflow job using the CLI
The following example demonstrates how to submit a DAG file to immediately run on CDE Airflow in Cloudera Data Engineering (CDE) using the command line interface (CLI).
cde airflow submit command is a quick and efficient way of
testing an Airflow job, as it spares you the task of creating and uploading
resources and job definitions before running the job, and cleaning up after running
This command is recommended only for Airflow DAGs that need to be run just once,
because the DAG is removed from Airflow at the end of the run. To manage Airflow
DAGs that need to be run more than once, or that contain schedules, use
job run instead of this command.
cde airflow submit <DAG python file> [--config-json <json-string>]* [--job-name <job name>]
To see the full command syntax and supported options, run
For example:To submit a job with a local DAG file:
cde airflow submit my-dag.py
When the job has been submitted the CLI displays the job run ID, waits for
the job to terminate, and returns an exit code of
1 for failure.