Learn how to create a CDE type connection to run jobs on other CDE Virtual
Clusters.
The following steps are for using the Airflow service provided with each CDE Virtual
Cluster. For information about using your own Airflow deployment, see Using Cloudera Data Engineering with an external
Apache Airflow deployment.
To create a connection to an existing CDE Virtual Cluster using the embedded Airflow
UI, perform the following steps:
-
In the Cloudera Data Platform (CDP) console, click the Data
Engineering tile. The CDE Home page displays.
-
Click Administration in the left navigation menu and
select the service containing the Virtual Cluster that you are using.
-
In the Virtual Clusters column, click Cluster
Details for the Virtual Cluster.
-
Click AIRFLOW UI.
-
From the Airflow UI, click the Connection link from the
Admin menu.
-
Click the plus sign to add a new record and fill the following fields:
- Conn Id: Create a unique connection identifier.
For example, cde_runtime_api.
- Conn Type: Select Cloudera Data
Engineering.
- Virtual Cluster API endpoint: Enter the target
Virtual Clusters Jobs API URL.
- CDP Access Key: Enter the CDP access key of the
account for running jobs on the CDE Virtual Cluster.
- CDP Private Key: Enter the CDP private key
associated to the CDP Access Key that you have entered.
- Extra: Extra arguments must in JSON format.
Available extra parameters are as follows:
-
Click Save.
