Creating an Airflow DAG using the Pipeline UI

With the CDE Pipeline UI, you can create multi-step pipelines with a combination of available operators.

This feature is available starting in CDE 1.16 in new Virtual Cluster installations.

Go to Jobs > Create Job.
Under Job details, select Airflow.

The UI refreshes, only Airflow-specific options remain.
Specify a name for the job.
Under DAG File select the Editor option.
Click Create.
You are redirected to the job Editor tab.
Build your Airflow pipeline.
- Drag and drop operators to the canvas from the left hand pane.
- When selecting an operator, you can configure it in the editor pane that opens up.
  On the Configure tab you can provide operator-specific settings. The Advanced tab allows you to make generic settings that are common to all operators, for example execution timeout or retries.
- Create dependencies between tasks by selecting them and drawing an arrow from one of the four nodes on their edges to another task. If the dependency is valid the task is highlighted in green. If invalid, it is highlighted in red.
- To modify DAG-level configuration, select Configurations on the upper right.
When you are done with building your pipeline, click Save.