Creating an Airflow DAG using the Pipeline UI

With the Cloudera Data Engineering Pipeline UI, you can create multi-step pipelines with a combination of available operators.

Go to Jobs > Create Job.
Under Job details, select Airflow.

The UI refreshes, only Airflow-specific options remain.
Specify a name for the job.
Under DAG File select the Editor option.
Click Create.
You are redirected to the job Editor tab.
note
The Cloudera Data Engineering Airflow Editor's Python Script (using Airflow Pure PythonOperator) is not automatically integrated with the Data Lake. As a result, when you run code through Create Job > Type: Airflow > Editor > Python Script > Run, the output does not have connection to the Data Lake resources.
To enable Airflow Pure PythonOperator integration with the Data Lake, manually configure the required connections and settings for the Data Lake connection.
Build your Airflow pipeline.
- Drag and drop operators to the canvas from the left hand pane.
- When selecting an operator, you can configure it in the editor pane that opens up.
  On the Configure tab you can provide operator-specific settings. The Advanced tab allows you to make generic settings that are common to all operators, for example execution timeout or retries.
- Create dependencies between tasks by selecting them and drawing an arrow from one of the four nodes on their edges to another task. If the dependency is valid the task is highlighted in green. If invalid, it is highlighted in red.
- To modify DAG-level configuration, select Configurations on the upper right.
When you are done with building your pipeline, click Save.