The following example demonstrates how to create an Airflow DAG in Cloudera Data
Engineering (CDE) using the command line interface (CLI).
-
Run the
cde job create
command as follows:
cde job create --name <job_name> --type airflow --dag-file <DAG_file> --mount-1-resource <your_DAG_resource> [other Airflow flags...]
<DAG_file>
- is a reference to a file within a CDE resource
To see the full command syntax and supported options, run cde job create
--help
.
-
Run
cde job describe
to verify that the job was created:
cde job describe --name <job_name>
-
If you want to update the job configuration, use the
cde job update
command.
For example, to change the number of
executors:
cde job update --name test_job
To see the full command syntax and supported options, run cde job update
--help
.
- To verify the updated configuration, run
cde job
describe
again:
cde job describe --name <job_name>