Creating and updating Apache Airflow jobs using the CLI

The following example demonstrates how to create an Airflow DAG in Cloudera Data Engineering (CDE) using the command line interface (CLI).

Make sure that you have downloaded the CLI client. For more information, see Using the Cloudera Data Engineering command line interface.
  1. Run the cde job create command as follows:
    cde job create --name <job_name> --type airflow --dag-file <DAG_file> --mount-1-resource <your_DAG_resource> [other Airflow flags...]
    <DAG_file>
    is a reference to a file within a CDE resource
    To see the full command syntax and supported options, run cde job create --help.
  2. Run cde job describe to verify that the job was created:
    cde job describe --name <job_name>
  3. If you want to update the job configuration, use the cde job update command.
    For example, to change the number of executors:
    cde job update --name test_job
    To see the full command syntax and supported options, run cde job update --help.
  4. To verify the updated configuration, run cde job describe again:
    cde job describe --name <job_name>