TLS and Kerberos Configuration for Cloudera Manager High Availability

Cloudera Manager supports TLS for encrypted network communications, and it supports integration with Kerberos for authentication (see Configuring TLS Security for Cloudera Manager and Configuring Authentication in Cloudera Manager for details). Configuring TLS- or Kerberos-enabled Cloudera Manager clusters for high availability requires the additional steps discussed below:

Example hostnames used throughout Configuring Cloudera Manager for High Availability With a Load Balancer are summarized in the table below.

Host description Example hostnames
Load balancer for Cloudera Manager Server CMSHostname
Cloudera Manager Server (primary) CMS1
Cloudera Manager Server (secondary) CMS2
Load balancer for Cloudera Management Service MGMTHostname
Cloudera Management Service (primary) MGMT1
Cloudera Management Service (secondary) MGMT2

TLS Considerations for Cloudera Manager High Availability

When successfully configured for high availability and for TLS, the Cloudera Manager Admin Console is accessed using the host name or IP address of the load balancer:

https://[CMSHostname]:7183

This assumes that the load balancer has been set up for TLS pass-through and that the Cloudera Manager Server host has been set up as detailed below.

Configure Load Balancers for TLS Pass-Through

As detailed in Configuring Cloudera Manager for High Availability With a Load Balancer, high availability for Cloudera Manager Server clusters requires secondary nodes that act as backups for the primary Cloudera Manager Server and Cloudera Management Service nodes, respectively. Only the primary nodes are active at any time, but if these fail, requests are redirected by a load balancer (CMSHostname or MGMTHostname) to the appropriate secondary node.

When the Cloudera Manager Server cluster is configured for TLS in addition to high availability, the load balancers must be configured for TLS pass-through—traffic from clients is not decrypted until it receives the actual server host system. Keep this in mind when you are Setting up the Load Balancer for your Cloudera Manager High Availability deployment.

Server Certificate Requirements for HA Deployments

TLS-enabled Cloudera Manager Server clusters require certificates that authenticate the host identity prior to encryption, as detailed in Level 2: Configuring TLS Verification of Cloudera Manager Server by the Agents. When deploying Cloudera Manager Server for high availability, however, the certificate must identify the load balancer and both primary and secondary nodes (rather than the primary host alone). That means you must create your certificate signing request (CSR) as follows:

  • Use the FQDN of the load balancer (for example, (CMSHostname) for the CN (common name).
  • Use the primary and secondary Cloudera Manager Server host names (for example, CMS1 and CMS2, respectively) for the SubjectAlternativeName (SAN) values.
To create a CSR using these example load balancer and server host names: 
keytool -genkeypair -alias loadBalProxyCMS  -keyalg RSA -keystore keystoreName.jks \
-keysize 2048 -dname "CN=CMSHostname, OU=Department, O=Example, \
L=City, ST=State, C=US" -storepass password \
-keypass password

keytool -certreq -alias loadBalProxyCMS -keystore keystoreName.jks -file sigRequestHA_LB_1.csr \
-storepass password -keypass password  -ext san=dns:CMS1,dns:CMS2
Alternatively, if the Cloudera Manager Server certificates on the hosts do not specify the load balancer name and SAN names, you can make the following change to the configuration. From the Cloudera Manager Admin Console, go to:
  • Administration > Ports and Addresses
  • Enter the FQDN of the load balancer in the Cloudera Manager Hostname Override
In addition to using correctly created certificates (or over-riding the hostname), you must:
  • Store the keystore and truststore in the same path on both primary and secondary Cloudera Manager Server hosts (CMS1, CMS2), or point to the same shared network mount point from each host.

Cloudera Manager Agent Host Requirements for HA Deployments

Cloudera Manager Server hosts can present their certificate to agents prior to encrypting the connection (TLS Level 2: see Enabling Server Certificate Verification on Cloudera Manager Agents for details). For a high availability deployment:
  • Use the same setting for verify_cert_file (in the /etc/cloudera-scm-agent/config.ini file) on each agent host system. To simplify the set, share the file path to verify_cert_file or copy the files manually as specified in the config file between MGMT1 and MGMT2.
Cloudera Manager Agent hosts can present certificates to requesting processes such as the Cloudera Manager Server prior to encryption (TLS Level 3: seeConfiguring Agent Certificate Authentication). For a high availability deployment:
  • Share the certificate and key for use by all Cloudera Manager Agent host systems on NFS, or copy them to the same path on both MGMT1 and MGMT2.

Kerberos Considerations for Cloudera Manager High Availability

As detailed in Creating Hosts for Primary and Secondary Servers, primary and secondary nodes that comprise a Cloudera Manager High Availability cluster must be configured the same (only one host is active at any given time). That means that if the cluster uses Kerberos for authentication, the Kerberos configuration on the primary and secondary nodes must also be the same.

Server Configuration Requirements for HA

When configuring high availability for Kerberos-enabled Cloudera Manager clusters, you must:

  • Install Kerberos client libraries in the same path on both primary (for example, CMS1) and secondary (CMS2) Cloudera Manager Server hosts .
  • Configure the /etc/krb5.conf file identically across the Cloudera Manager Server and Cloudera Management Service hosts (CMS1, CMS2, MGMT1, MGMT2).
  • If the Cloudera Manager Server primary host (CMS1) is configured to store the Cloudera Manager Server KDC access credentials in /etc/cloudera-scm-server, use this same path on the secondary host (CMS2).

Re-Generate Kerberos Credentials

Configuring the Cloudera Management Service for high availability using an existing Cloudera Manager Server cluster (as discussed in Installing the Primary) results in the Cloudera Management Service not starting, as shown here:



This is expected. To resolve, re-generate the Kerberos credentials for the roles:

  1. Log in to the Cloudera Manager Admin Console.
  2. Select Administration > Kerberos > Credentials > Generate Credentials.