Installing Cloudera Manager with High Availability

Steps to install Cloudera Manager with high availability.

  1. Provision an external database for use by Cloudera Manager.
  2. Provision two hosts for Cloudera Manager.
  3. Repeat the steps to install Cloudera Manager on each of the hosts to be used for Cloudera Manager high availability. Skip these auto-TLS steps on the passive host. Do not start Cloudera Manager on any hosts until you complete the following:

    1. Configure both the active and passive Cloudera Manager instances to connect to the same database. You configure this by running the scm_prepare_database.sh script on each Cloudera Manager host. Specify the database host with the --host option.

      1. On each Cloudera Manager host, edit the following file: /etc/default/cloudera-scm-server

      2. Find the line that begins with export CMF_JAVA_OPTS and add the following parameter:
        -Dcom.cloudera.cmf.haMode=true
      3. Find the line that begins with export CMF_SERVER_ARGS and insert the following inside the quotes: --ha-priority <number>

      The --ha-priority sets the priority of the server. The number has to be a non zero positive integer. The lower the number is, the higher priority the server has. Thus, the active server should have a smaller “ha-priority” number than the passive server. When the hardware of the previous running host gets replaced, make sure the new active server always has a higher priority (smaller ha-priority number) than the passive server. To start with, the suggested values are ‘10’ for the active server, and ‘20’ for the passive server. Increment by 10 for each additional passive server.

      For example:
      export CMF_SERVER_ARGS=”--ha-priority 10”
      export CMF_JAVA_OPTS="-Xmx2G -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError  -XX:HeapDumpPath=/tmp -Dcom.cloudera.cmf.haMode=true
    2. Save the file

  4. Start one instance of Cloudera Manager and wait for the Cloudera Manager server to come online.

  5. Start the Load Balancer.

  6. Configure the Cloudera Manager Hostname Override:

    1. Log in to the Cloudera Manager Admin Console, and go to Administration > Settings.

    2. Select Ports and Addresses.

    3. Set the Cloudera Manager Hostname Override parameter to contain the fully qualified domain name of the load balancer.

  7. If you have enabled TLS for Cloudera Manager, do the following:

    1. Locate the Verify Agent Hostname Against Certificate configuration parameter, and uncheck it. (Go to Administration > Settings and search for this parameter.)
    2. Ensure that the CA certificate signing the load balancer’s TLS certificate is trusted by Cloudera Manager server.
    3. The CA certificate signing the load balancer’s TLS certificate must be specified in the file referenced by the Cloudera Manager TLS/SSL Trust Store File configuration parameter (Go to Administration > Settings > Security and search for this parameter.)
  8. Restart the active instance of Cloudera Manager Server.
  9. CM agents on all hosts must point to load balancer for communicating with CM servers.
    If you are adding high availability to an existing set of hosts managed by Cloudera Manager, configure the Cloudera Manager agents on all managed hosts with the fully-qualified domain name of the load balancer.

    If your host provisioning system automatically adds new hosts to Cloudera Manager without using the Cloudera Manager Add Host wizard, ensure the above server_host line in the /etc/cloudera-scm-agent/config.ini is configured with the fully-qualified domain name of the load balancer. This may be needed in systems using pre-created host images.

  10. If you have enabled TLS for Cloudera Manager, the CA certificate signing the load balancer’s TLS certificate must be specified. On all Cloudera Manager-managed hosts, add the following to the /etc/cloudera-scm-agent/config.ini file:
    verify_cert_file=load_balancer_ca_cert.pem

    (Replace load_balancer_ca_cert.pem with the file containing the certificate.)

  11. Perform rotation of Auto-TLS certificates (See Rotate Auto-TLS Certificate Authority and Host Certificates) with location parameter set to blank to store the certificates in DB instead of file system. This is important to ensure that both the hosts are able to use the auto TLS certificates after rotation.
  12. Restart the Cloudera Manager agents by running the following command on all managed hosts:
    sudo service cloudera-scm-agent restart
  13. Start the passive instance of the Cloudera Manager server.
  14. Continue with cluster installation and wait for the new cluster to start up.
  15. Verify that both Cloudera Manager hosts are showing good health in the Cloudera Manager Admin Console. (Go to Hosts > All Hosts to check the status of the hosts of the Cloudera Manager Servers.)

Enabling Cloudera Manager High Availability in an Existing Environment

You can enable Cloudera Manager High Availability (HA) in an existing deployment where an active Cloudera Manager Server and external database are already running. Each step in this procedure includes a label indicating where you must perform the action: [Active CM Server], [Standby CM Server], [Both], or [All Managed Hosts].
  1. [Both] Verify external database connectivity and hosting.
    Ensure that your external database is reachable over the network from both the active and standby Cloudera Manager hosts.
  2. [Standby CM Server] Provision and install Cloudera Manager on the standby host.

    Provision a new host for the standby Cloudera Manager Server. Install the exact same version of Cloudera Manager on the standby host as the version running on the active host. Skip any Auto-TLS configuration steps on the standby host, and do not start the Cloudera Manager Server yet.

  3. [Both] Copy server configurations, parcel repositories, and custom service descriptors.

    Copy the existing Cloudera Manager Server configuration files, local parcel repository (/opt/cloudera/parcel-repo by default), and Custom Service Descriptors (CSDs) (/opt/cloudera/csd by default) from the active Cloudera Manager Server host to the standby Cloudera Manager Server host. Ensure that file ownership and permissions match the active host.

  4. [Standby CM Server] Point the standby Cloudera Manager Server to the existing database.

    Configure the standby Cloudera Manager Server instance to connect to your existing external database by running the scm_prepare_database.sh script (or by copying /etc/cloudera-scm-server/db.properties from the active host). If you run scm_prepare_database.sh, specify the existing database host with the --host option.

  5. [Both] Enable HA mode and set server priorities.
    On each Cloudera Manager Server host, edit the /etc/default/cloudera-scm-server file:
    1. Locate the line that begins with export CMF_JAVA_OPTS and add the following parameter:
      -Dcom.cloudera.cmf.haMode=true
    2. Locate the line that begins with export CMF_SERVER_ARGS and add the --ha-priority <number> parameter inside the quotes. Assign a lower number (higher priority) to the active server (10) and a higher number to the standby server (20).
      For example, on the active server:
      export CMF_SERVER_ARGS="--ha-priority 10"
      
      export CMF_JAVA_OPTS="-Xmx2G -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Dcom.cloudera.cmf.haMode=true"
    3. Save the file on both hosts.
  6. [Active CM Server] Configure the Cloudera Manager Hostname Override.
    Start your load balancer, then update the hostname override setting in the Cloudera Manager Admin Console:
    1. Log in to the Cloudera Manager Admin Console and navigate to Administration > Settings.
    2. Select Ports and Addresses.
    3. Set the Cloudera Manager Hostname Override parameter to the fully qualified domain name (FQDN) of the load balancer.
  7. [Active CM Server] Update Transport Layer Security (TLS) settings (if TLS is enabled).
    If TLS is enabled for Cloudera Manager, complete the following steps in the Cloudera Manager Admin Console:
    1. Navigate to Administration > Settings, locate the Verify Agent Hostname Against Certificate parameter, and clear the checkbox.
    2. Navigate to Administration > Settings > Security and ensure that the Certificate Authority (CA) certificate signing the load balancer TLS certificate is included in the truststore referenced by the Cloudera Manager TLS/SSL Trust Store File parameter.
  8. [Active CM Server] Restart the active Cloudera Manager Server.

    Restart the Cloudera Manager Server service on the active host so that the HA mode, priority, and load balancer settings take effect.

  9. [All Managed Hosts] Point Cloudera Manager Agents to the load balancer.
    Update the agent configuration on all cluster hosts to communicate through the load balancer:
    1. Open /etc/cloudera-scm-agent/config.ini and update the server_host property to the FQDN of the load balancer:
      server_host=<load_balancer_fqdn>
    2. If TLS is enabled, add the load balancer CA certificate path to /etc/cloudera-scm-agent/config.ini:
      verify_cert_file=<load_balancer_ca_cert.pem>
    3. If you use Auto-TLS, rotate the Auto-TLS certificates (with the location parameter left blank) so that certificates are stored in the shared database rather than the local file system.
    4. Restart the Cloudera Manager Agent on all managed hosts:
      sudo service cloudera-scm-agent restart
  10. [Standby CM Server] Start the standby Cloudera Manager Server.
    Start the Cloudera Manager Server service on the standby host:
    sudo service cloudera-scm-server start
  11. [Active CM Server] Verify high availability health.

    Log in to the Cloudera Manager Admin Console, navigate to Hosts > All Hosts, and verify that both the active and standby Cloudera Manager Server hosts report good health.

Replacing Cloudera Manager Hosts

Primary/secondary node fails:
  1. Set up a new Cloudera Manager server node as per above steps
  2. Update the load balancer configuration to update the host name of replaced host.
  3. Ensure that the priority of active host remains higher (smaller value of the --ha-priority property) than the passive host.
  4. Ensure the Cloudera Manager server is running on the host before restarting load balancer.
  5. Restart load balancer server.
Load Balancer fails
  1. Set up a new load balancer using the existing configuration.
  2. Ensure that the load balancer endpoint remains the same as it is configured in the Cloudera Manager server and agents.