Register an HDP cluster reachable through private IP address

To ensure optimum security, clusters within the customer environment are not accessible for communication. They have private IP addresses and cannot be accessed outside the firewall. However, to add your cluster to the CDP, a communication line needs to be established.

A reverseSSH tunnel solves the problem by establishing a tunnel from the cluster to CDP. You must download and install AutoSSH and the connectivity install scripts to establish a secure two-way communication channel. The AutoSSH ensures that the connectivity is stable. The connectivity scripts and their installation ensure safe connectivity and communication.

Prerequisites

  • HDP clusters must be managed by Ambari. Clusters that are not managed by Ambari cannot be registered to CDP.
  • HDP clusters must include Knox.
  • HDP clusters must include Ranger policy settings
  • LDAP/AD must be set up and synced in Ambari

    LDAP settings are automatically detected from the default topology setup in Knox. If the default topology does not have the LDAP setup, you will be asked to provide another topology name where you have configured the LDAP. If that topology has LDAP the setup continues. If the LDAP is not configured, you will receive an error message.

  • Kerberos must be enabled on the HDP cluster and the LDAP/AD must be set up in the Kerberos authentication so that the same set of LDAP/AD credentials can be used to access Ambari APIs as well as Beacon APIs
  • All clusters must meet the requirements identified in Prerequisites for adding classic clusters.

The process to register an HDP cluster using a reverseSSH tunnel is as follows:

  1. Log in to CDP Management Console.
  2. Click Classic Clusters in the left navigation panel.

    The Classic Clusters page appears.

  3. If you are a first time user, under Step 1 in the Register Classic Cluster wizard, click GET STARTED. Classic Cluster then displays the Cluster Details dialog box.

    If you are not a first time user, click the ADD CLUSTER button on the right side of the listing page.

  4. Click HDP.
  5. Provide the following connectivity information for your new cluster, then click CONNECT:
    • IP Address
    • Port
    • Data center

    After Classic Cluster establishes connection, it will highlight Step 2 in the Register Classic Cluster wizard.

  6. Start the download and installation process for the SSH connectivity files by clicking the Files button in Step 2 of the wizard.
  7. Follow the instructions in the Setup Connectivity Client dialog box, downloading and installing the ccm-autosssh-client and ssh_tunnel_setup_files onto your new cluster.
  8. Copy the files to your Knox node or the Knox proxy host in the cluster.
  9. Run the install.sh script.

    ./install.sh

  10. Enter the following information as the install script prompts for it:
    • Enter Ambari URL (http(s)://host:[port]):
    • Enter Ambari Username:
    • Enter Ambari Password:
  11. If Knox is not installed on a proxy server, proceed to Step 13.

    Classic Cluster sets up the topology for the Knox server and establishes the reverseSSH tunnel.

  12. If Knox is installed on a proxy server, Classic Cluster displays the following message:

    We discovered that your Knox is installed in HA mode. Please confirm if this node is your proxy node (yes/no):

    Enter yes.

    Classic Cluster generates XML content that you will need to add to your Knox hosts. Classic Cluster also displays three steps you must perform on all of your Knox hosts:

    1. Copy the generated XML to /usr/hdp/current/knox-server/conf/topologies/cdp_default.xml.

      For example:

      <?xml version ='1.0' encoding='utf8'?>
      <topology> 
        <gateway>       
          <provider>  
            <role>authentication</role>     
            <name>ShiroProvider</name>   
            <enabled>true</enabled>   
            <param>   
              <name>sessionTimeout</name>  
              <value>30</value>
              .
              .
              .
      </topology>>
                    
    2. Run chown knox:knox cdp_default.xml.
    3. Check the Knox logs/deployment directory to verify that the cdp_default topology is deployed.
    4. After you have completed Steps a through c on all of your Knox hosts, press Enter to continue.

      This sets up the topology for the Knox server or Knox proxy host and establishes the reverseSSH tunnel.

  13. Classic Cluster starts checking the connectivity with the cluster. When the connectivity is successful, proceed to Step 3 in the wizard.

    If the connection attempts fail or if there is an error in the connectivity, Classic Cluster displays troubleshooting information in Step 2 of the Registration wizard. Follow the troubleshooting information to fix the connectivity error, then click Test connection.

  14. Click Register in Step 3 of the Registration wizard.
  15. In the Cluster Details dialog box, provide the username and password to access the cluster, then click CONNECT.

    The user should have Admin access to the customer cluster services.

  16. Finish registering the cluster by providing the following information.
    • Cluster Location
    • Data Center
    • Tags (optional)
    • Description (optional)

    If LDAP is not set up on the default topology, the system will ask for the following additional information:

    Enter knox topology name that contains LDAP setup:

  17. Click ADD.