Learn how to create a Cloudera Data Engineering type connection to run jobs on other Cloudera Data Engineering Virtual
Clusters.
The following steps are for using the Airflow service provided with each Cloudera Data Engineering Virtual
Cluster. For information about using your own Airflow deployment, see Using Cloudera Data Engineering with an external
Apache Airflow deployment.
To create a connection to an existing Cloudera Data Engineering Virtual Cluster using the embedded Airflow
UI, perform the following steps:
-
In the Cloudera console, click the Data
Engineering tile. The Cloudera Data Engineering Home page displays.
-
Click Administration in the left navigation menu and
select the service containing the Virtual Cluster that you are using.
-
In the Virtual Clusters column, click Cluster
Details for the Virtual Cluster.
-
Click AIRFLOW UI.
-
From the Airflow UI, click the Connection link from the
Admin menu.
-
Click the plus sign to add a new record and fill the following fields:
- Conn Id: Create a unique connection identifier.
For example, cde_runtime_api.
- Conn Type: Select Cloudera Data
Engineering.
- Virtual Cluster API endpoint: Enter the target
Virtual Clusters Jobs API URL.
- Cloudera Access Key: Enter the Cloudera access key of the
account for running jobs on the Cloudera Data Engineering Virtual Cluster.
- Cloudera Private Key: Enter the Cloudera private key
associated to the Cloudera Access Key that you have entered.
- Extra: Extra arguments must in JSON format.
Available extra parameters are as follows:
-
Click Save.
