Adding an HDP classic cluster

You must register HDP clusters with Cloudera before you can use them with Cloudera Public Cloud services and components.

About this task

To ensure optimum security, clusters within the customer environment are not accessible for communication: They have private IP addresses and cannot be accessed outside the firewall. To add your cluster to Cloudera, a communication line needs to be established.

The Cluster Connectivity Manager inverted proxy solves the problem by establishing a connection from the on-premise cluster to Cloudera. You must download and install the jumpgate agent and the connectivity install scripts to establish a secure two-way communication channel. The jumpgate agent ensures that the connectivity is stable. The connectivity scripts and their installation ensure safe connectivity and communication.

Before you begin

HDP clusters must be managed by Ambari. HDP clusters that are not managed by Ambari cannot be registered to Cloudera.
HDP clusters must include Knox.
HDP clusters must include Ranger policy settings
LDAP/AD must be set up and synced in Ambari.
LDAP settings are automatically detected from the default topology setup in Knox. If the default topology does not have the LDAP setup, you will be asked to provide another topology name where you have configured the LDAP. If that topology has LDAP, the setup continues. If the LDAP is not configured, you will receive an error message.
Kerberos must be enabled on the HDP cluster and the LDAP/AD must be set up in the Kerberos authentication so that the same set of LDAP/AD credentials can be used to access Ambari APIs as well as Beacon APIs.
All clusters must meet the requirements identified in Prerequisites for adding classic clusters.

Steps

The process to register an HDP cluster is as follows:

Log in to Cloudera and navigate to the Cloudera Management Console.
Click Classic Clusters in the left navigation panel.
Click Add Cluster.

CDP displays the Cluster Details dialog box.
If you are a first time user, under Step 1 in the Register Classic Cluster wizard, click GET STARTED. If you are not a first time user, click the ADD CLUSTER button on the right side of the listing page.

CDP displays the Cluster Details dialog box.
Select HDP.
Provide the following connectivity information for your new cluster:
1. Knox IP Address
2. Knox port
3. Data center
4. Click CONNECT
Step 1 might take up to 5 minutes if you are adding the cluster for the first time. After Classic Cluster establishes connection, Cloudera will highlight Step 2 in the Register Classic Cluster wizard.
Start the download and installation process for the connectivity files by clicking the Files button in Step 2 of the wizard.
Follow the instructions in the Setup Connectivity Client dialog box. You need to download the jumpgate-agent rpm file and the cluster_connectivity_setup_files zip files and copy them to your Knox node. Or if your Knox is running in HA mode, you need to copy the files to the Knox proxy host in the cluster:
1. Download the jumpgate-agent RPM and cluster_connectivity_setup_files.zip.
2. In the command line interface, copy the two files to the Knox or Knox proxy host.
3. SSH to the host.
4. Install the jumpgate-agent rpm using:
```
yum --nogpgcheck localinstall < downloaded-jumpgate-agent-rpm >
```
1. Unzip the cluster_connectivity_setup_files file. Inside this zip file there is a script install.sh.
2. Run install.sh by using ./install.sh command.
3. Check service status to see if the agent has been connected:
```
systemctl status jumpgate-agent.service 
```
Enter the following information as the install script prompts for it:
1. Enter Ambari URL (http(s)://host:[port]):
2. Enter Ambari Username:
3. Enter Ambari Password:
If Knox is not installed on a proxy server, proceed to Step 12. Classic Cluster sets up the topology for the Knox server and establishes the connection.
If Knox is installed on a proxy server, Classic Cluster displays the following message:We discovered that your Knox is installed in HA mode. Please confirm if this node is your proxy node (yes/no):Enter yes. Classic Cluster generates XML content that you will need to add to your Knox hosts. Classic Cluster also displays three steps you must perform on all of your Knox hosts:
1. Copy the generated XML to /usr/hdp/current/knox-server/conf/topologies/cdp_default.xml
2. Run chown knox:knox cdp_default.xml
3. Check the Knox logs/deployment directory to verify that the cdp_default topology is deployed.
4. After you have completed steps a through c on all of your Knox hosts, type Enter to continue.This sets up the topology for the Knox server or Knox proxy host and establishes the connection.
On the Classic Clusters page, click Test Connection in the Step 2 pane to verify whether the connection is successful.
Cloudera starts checking the connectivity with the HDP cluster. When the connectivity is successful, proceed to Step 3 in the wizard.
If the connection attempts fail or if there is an error in the connectivity, Cloudera displays troubleshooting information in Step 2 of the registration wizard. Follow the troubleshooting information to fix the connectivity error, then click Test connection.
note
After you download the files, the cluster_connectivity_setup files download is disabled. At this point, you can regenerate the cluster connectivity setup files using the regenerate files option. This option comes in handy if you lose the files previously downloaded before you can set the cluster connectivity in the cluster.
Click Register in Step 3 of the registration wizard.
In the Cluster Details dialog box, provide the username and password to access the cluster, then click CONNECT.
The user should have admin access to the cluster services.
Finish registering the cluster by providing the following information.
1. Cluster Location
2. Data Center
3. Tags (optional)
4. Description (optional)
If LDAP is not set up on the default topology, the system will ask for the following additional information: Enter knox topology name that contains LDAP setup.
Click ADD.

Result

Once the cluster has been registered, you can use it with Cloudera.