Creating a connection to run jobs on other Cloudera Data Engineering Virtual Clusters

Learn how to create a Cloudera Data Engineering type connection to run jobs on other Cloudera Data Engineering Virtual Clusters.

The following steps are for using the Airflow service provided with each Cloudera Data Engineering Virtual Cluster. For information about using your own Airflow deployment, see Using Cloudera Data Engineering with an external Apache Airflow deployment.

To create a connection to an existing Cloudera Data Engineering Virtual Cluster using the embedded Airflow UI, perform the following steps:

In the Cloudera console, click the Data Engineering tile. The Cloudera Data Engineering Home page displays.
Click Administration in the left navigation menu and select the service containing the Virtual Cluster that you are using.
In the Virtual Clusters column, click Cluster Details for the Virtual Cluster.
Click AIRFLOW UI.
From the Airflow UI, click the Connection link from the Admin menu.
Click the plus sign to add a new record and fill the following fields:
- Conn Id: Create a unique connection identifier. For example, cde_runtime_api.
- Conn Type: Select Cloudera Data Engineering.
- Virtual Cluster API endpoint: Enter the target Virtual Clusters Jobs API URL.
- Cloudera Access Key: Enter the Cloudera access key of the account for running jobs on the Cloudera Data Engineering Virtual Cluster.
- Cloudera Private Key: Enter the Cloudera private key associated to the Cloudera Access Key that you have entered.
- Extra: Extra arguments must in JSON format. Available extra parameters are as follows:
  - Proxy: Optional. Translates to https_proxy/HTTPS_PROXY environment variables. The default value is None.
  - Region: Optional. Cloudera Control Plane region ("us-west-1", "eu-1" or "ap-1") is inferred automatically, if not specified.
    For more information about the available extra arguments, see the GitHub page.
Click Save.

We want your opinion