Creating a basic Airflow pipeline using CDE CLI

By creating a basic pipeline in Cloudera Data Engineering (CDE) using the CLI, you can create multi-step pipelines with a combination of available operators.

To create a basic pipeline in CDE, you must upload the Airflow (Directed Acyclic Graph) DAG to a CDE resource and create a CDE Airflow job from this DAG.

In the CDE CLI, run the following command:
cde resource create --name my_pipeline_resource
cde resource upload --name my_pipeline_resource --local-path
cde job create --name my_pipeline --type airflow --dag-file --mount-1-resource my_pipeline_resource