With the CDE Pipeline UI, you can create multi-step pipelines with a combination of
available operators.
This feature is available in CDE 1.16 in new Virtual Cluster installations only.
The Pipeline UI has been enabled during Virtual Cluster
creation.
Go to Jobs > Create Job.
Under Job details, select
Airflow.
The UI refreshes, only Airflow-specific options remain.
Specify a name for the job.
Under DAG File select the Editor
option.
Click Create.
You are redirected to the job Editor
tab.
Build your Airflow pipeline.
Drag and drop operators to the canvas from the left hand pane.
When selecting an operator, you can configure it in the editor pane that
opens up.
On the Configure tab you can provide
operator-specific settings. The Advanced tab
allows you to make generic settings that are common to all
operators, for example execution timeout or retries.
Create dependencies between tasks by selecting them and drawing an arrow
from one of the four nodes on their edges to another task. If the
dependency is valid the task is highlighted in green. If invalid, it is
highlighted in red.
To modify DAG-level configuration, select
Configurations on the upper right.
When you are done with building your pipeline, click
Save.
