Running a Airflow job using the CLI

The following example demonstrates how to run a Cloudera Data Engineering (CDE) Airflow job using the command line interface (CLI).

Make sure that the job has been created and all necessary resources have been created and uploaded.

Using the cde job run requires more preparation on the target environment compared to the cde airflow submit command. Whereas cde airflow submit is a quick and efficient way of testing an Airflow job during development, cde job run is suited for production environments where a job is to be run multiple times, therefore removing resources and job definitions after every job run is neither necessary, nor viable.

To run an Airflow job, run the following command:
cde job run --name <job name> [--config <key=value>]* [--wait]

Airflow configs provided at job run time will override the corresponding job configs.

By default the command returns the job run ID as soon as the job has been submitted.

Optionally, you can use the --wait switch to wait until the job run ends and returns a non-zero exit code if the job run was not successful.