Creating an Airflow DAG using the Pipeline UI
With the Cloudera Data Engineering Pipeline UI, you can create multi-step pipelines with a combination of available operators.
-
Go to Jobs > Create Job.
Under Job details, select Airflow.The UI refreshes, only Airflow-specific options remain.
- Specify a name for the job.
- Under DAG File select the Editor option.
-
Click Create.
You are redirected to the job Editor tab.
-
Build your Airflow pipeline.
- Drag and drop operators to the canvas from the left hand pane.
- When selecting an operator, you can configure it in the editor pane that
opens up.
On the Configure tab you can provide operator-specific settings. The Advanced tab allows you to make generic settings that are common to all operators, for example execution timeout or retries.
- Create dependencies between tasks by selecting them and drawing an arrow from one of the four nodes on their edges to another task. If the dependency is valid the task is highlighted in green. If invalid, it is highlighted in red.
- To modify DAG-level configuration, select Configurations on the upper right.
- When you are done with building your pipeline, click Save.
