Installing Cloudera Manager with High Availability
Steps to install Cloudera Manager with high availability.
- Provision an external database for use by Cloudera Manager.
- Provision two hosts for Cloudera Manager.
-
Repeat the steps to install Cloudera Manager on each of the hosts to be used for Cloudera Manager high availability. Skip these auto-TLS steps on the passive host. Do not start Cloudera Manager on any hosts until you complete the following:
-
Configure both the active and passive Cloudera Manager instances to connect to the same database. You configure this by running the
scm_prepare_database.shscript on each Cloudera Manager host. Specify the database host with the--hostoption. -
-
On each Cloudera Manager host, edit the following file:
/etc/default/cloudera-scm-server - Find the line that begins with
export CMF_JAVA_OPTSand add the following parameter:-Dcom.cloudera.cmf.haMode=true - Find the line that begins with
export CMF_SERVER_ARGSand insert the following inside the quotes:--ha-priority <number>
The
--ha-prioritysets the priority of the server. The number has to be a non zero positive integer. The lower the number is, the higher priority the server has. Thus, the active server should have a smaller “ha-priority” number than the passive server. When the hardware of the previous running host gets replaced, make sure the new active server always has a higher priority (smaller ha-priority number) than the passive server. To start with, the suggested values are ‘10’ for the active server, and ‘20’ for the passive server. Increment by 10 for each additional passive server.For example:export CMF_SERVER_ARGS=”--ha-priority 10” export CMF_JAVA_OPTS="-Xmx2G -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Dcom.cloudera.cmf.haMode=true -
-
Save the file
-
-
Start one instance of Cloudera Manager and wait for the Cloudera Manager server to come online.
-
Start the Load Balancer.
-
Configure the Cloudera Manager Hostname Override:
-
Log in to the Cloudera Manager Admin Console, and go to Administration > Settings.
-
Select Ports and Addresses.
-
Set the Cloudera Manager Hostname Override parameter to contain the fully qualified domain name of the load balancer.
-
-
If you have enabled TLS for Cloudera Manager, do the following:
- Locate the Verify Agent Hostname Against Certificate configuration parameter, and uncheck it. (Go to Administration > Settings and search for this parameter.)
- Ensure that the CA certificate signing the load balancer’s TLS certificate is trusted by Cloudera Manager server.
- The CA certificate signing the load balancer’s TLS certificate must be specified in the file referenced by the Cloudera Manager TLS/SSL Trust Store File configuration parameter (Go to and search for this parameter.)
- Restart the active instance of Cloudera Manager Server.
- CM agents on all hosts must point to load balancer for
communicating with CM servers.If you are adding high availability to an existing set of hosts managed by Cloudera Manager, configure the Cloudera Manager agents on all managed hosts with the fully-qualified domain name of the load balancer.
If your host provisioning system automatically adds new hosts to Cloudera Manager without using the Cloudera Manager Add Host wizard, ensure the above
server_hostline in the/etc/cloudera-scm-agent/config.iniis configured with the fully-qualified domain name of the load balancer. This may be needed in systems using pre-created host images. -
If you have enabled TLS for Cloudera Manager, the CA certificate signing the load balancer’s TLS certificate must be specified. On all Cloudera Manager-managed hosts, add the following to the
/etc/cloudera-scm-agent/config.inifile:verify_cert_file=load_balancer_ca_cert.pem(Replace load_balancer_ca_cert.pem with the file containing the certificate.)
- Perform rotation of Auto-TLS certificates (See Rotate Auto-TLS Certificate Authority and Host Certificates) with location parameter set to blank to store the certificates in DB instead of file system. This is important to ensure that both the hosts are able to use the auto TLS certificates after rotation.
-
Restart the Cloudera Manager agents by running the following command on all managed hosts:
sudo service cloudera-scm-agent restart - Start the passive instance of the Cloudera Manager server.
- Continue with cluster installation and wait for the new cluster to start up.
- Verify that both Cloudera Manager hosts are showing good health in the Cloudera Manager Admin Console. (Go to to check the status of the hosts of the Cloudera Manager Servers.)
Enabling Cloudera Manager High Availability in an Existing Environment
-
- [Both] Verify external database connectivity and hosting.
-
Ensure that your external database is reachable over the network from both the active and standby Cloudera Manager hosts.
-
- [Standby CM Server] Provision and install Cloudera Manager on the standby host.
-
Provision a new host for the standby Cloudera Manager Server. Install the exact same version of Cloudera Manager on the standby host as the version running on the active host. Skip any Auto-TLS configuration steps on the standby host, and do not start the Cloudera Manager Server yet.
-
- [Both] Copy server configurations, parcel repositories, and custom service descriptors.
-
Copy the existing Cloudera Manager Server configuration files, local parcel repository (
/opt/cloudera/parcel-repoby default), and Custom Service Descriptors (CSDs) (/opt/cloudera/csdby default) from the active Cloudera Manager Server host to the standby Cloudera Manager Server host. Ensure that file ownership and permissions match the active host.
-
- [Standby CM Server] Point the standby Cloudera Manager Server to the existing database.
-
Configure the standby Cloudera Manager Server instance to connect to your existing external database by running the
scm_prepare_database.shscript (or by copying/etc/cloudera-scm-server/db.propertiesfrom the active host). If you runscm_prepare_database.sh, specify the existing database host with the--hostoption.
-
- [Both] Enable HA mode and set server priorities.
-
On each Cloudera Manager Server host, edit the
/etc/default/cloudera-scm-serverfile:- Locate the line that begins with
export CMF_JAVA_OPTSand add the following parameter:-Dcom.cloudera.cmf.haMode=true - Locate the line that begins with
export CMF_SERVER_ARGSand add the--ha-priority <number>parameter inside the quotes. Assign a lower number (higher priority) to the active server (10) and a higher number to the standby server (20).For example, on the active server:export CMF_SERVER_ARGS="--ha-priority 10" export CMF_JAVA_OPTS="-Xmx2G -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Dcom.cloudera.cmf.haMode=true" - Save the file on both hosts.
- Locate the line that begins with
-
- [Active CM Server] Configure the Cloudera Manager Hostname Override.
-
Start your load balancer, then update the hostname override setting in the Cloudera Manager Admin Console:
- Log in to the Cloudera Manager Admin Console and navigate to .
- Select Ports and Addresses.
- Set the Cloudera Manager Hostname Override parameter to the fully qualified domain name (FQDN) of the load balancer.
-
- [Active CM Server] Update Transport Layer Security (TLS) settings (if TLS is enabled).
-
If TLS is enabled for Cloudera Manager, complete the following steps in the Cloudera Manager Admin Console:
- Navigate to , locate the Verify Agent Hostname Against Certificate parameter, and clear the checkbox.
- Navigate to and ensure that the Certificate Authority (CA) certificate signing the load balancer TLS certificate is included in the truststore referenced by the Cloudera Manager TLS/SSL Trust Store File parameter.
-
- [Active CM Server] Restart the active Cloudera Manager Server.
-
Restart the Cloudera Manager Server service on the active host so that the HA mode, priority, and load balancer settings take effect.
-
- [All Managed Hosts] Point Cloudera Manager Agents to the load balancer.
-
Update the agent configuration on all cluster hosts to communicate through the load balancer:
- Open
/etc/cloudera-scm-agent/config.iniand update theserver_hostproperty to the FQDN of the load balancer:server_host=<load_balancer_fqdn> - If TLS is enabled, add the load balancer CA certificate path to
/etc/cloudera-scm-agent/config.ini:verify_cert_file=<load_balancer_ca_cert.pem> - If you use Auto-TLS, rotate the Auto-TLS certificates (with the location parameter left blank) so that certificates are stored in the shared database rather than the local file system.
- Restart the Cloudera Manager Agent on all managed
hosts:
sudo service cloudera-scm-agent restart
- Open
-
- [Standby CM Server] Start the standby Cloudera Manager Server.
-
Start the Cloudera Manager Server service on the standby host:
sudo service cloudera-scm-server start
-
- [Active CM Server] Verify high availability health.
-
Log in to the Cloudera Manager Admin Console, navigate to , and verify that both the active and standby Cloudera Manager Server hosts report good health.
Replacing Cloudera Manager Hosts
- Set up a new Cloudera Manager server node as per above steps
- Update the load balancer configuration to update the host name of replaced host.
- Ensure that the priority of active host remains higher (smaller
value of the
--ha-priorityproperty) than the passive host. - Ensure the Cloudera Manager server is running on the host before restarting load balancer.
- Restart load balancer server.
- Set up a new load balancer using the existing configuration.
-
Ensure that the load balancer endpoint remains the same as it is configured in the Cloudera Manager server and agents.
