Troubleshooting TLS/SSL Issues in Cloudera Manager

This topic contains instructions for diagnosing and fixing issues you might face on a TLS-enabled cluster.

Inspecting Cloudera Manager Connectivity with OpenSSL

The openssl tool can be run from the host that is running the Cloudera Manager Agent or client service that should be inspected for connectivity issues. You should also test whether the certificate in use by the host is recognized by a trusted CA during the TLS/SSL negotiation.

Use the following command to inspect the connection. 
$ openssl s_client -connect [host.fqdn.name]:[port]
For example: 
$ openssl s_client -connect test1.sec.cloudera.com:7183
A return code 0 means openssl was able to establish trust of the server through its library of trusted public CAs. If the certificate was self-signed (recommended only on test clusters) or provided by a private CA, it might be necessary to add the private CA or self-signed certificate to the truststore using the openssl command. Adding the path to the root CA, -CAfile </path/to/root-ca.pem>, allows openssl to verify your self-signed or private CA-signed certificate as follows: 
$ openssl s_client -connect test1.sec.cloudera.com:7183 -CAfile \
/opt/cloudera/security/CAcerts/RootCA.pem
Note that providing only the Root CA certificate is necessary to establish trust for this test. The result from the command is successful when you see the return code 0 as follows: 
...
 Verify return code: 0 (ok)
---
By default, the Cloudera Manager Server writes logs to the /etc/cloudera-scm-server/cloudera-scm-server.log file on startup. Successful start of the server process with the certificate will show logs similar to the following: 
2014-10-06 21:33:47,515 INFO WebServerImpl:org.mortbay.log: jetty-6.1.26.cloudera.2 
2014-10-06 21:33:47,572 INFO WebServerImpl:org.mortbay.log: Started SslSelectChannelConnector@0.0.0.0:7183 
2014-10-06 21:33:47,573 INFO WebServerImpl:org.mortbay.log: Started SelectChannelConnector@0.0.0.0:7180 
2014-10-06 21:33:47,573 INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.

Uploading Diagnostic Bundles to Cloudera Fails

By default, Cloudera Manager uses HTTPS to upload diagnostic bundles to the Cloudera Support server at cops.cloudera.com. These uploads will fail if Cloudera Manager cannot confirm the authenticity of the Cloudera Support server. The Support server authenticates itself to Cloudera Manager by presenting a certificate signed by a public Certificate Authority (CA). However, if you have previously enabled Level 3 TLS Authentication for Cloudera Manager, the Cloudera Manager truststore may not contain certificates signed by public CAs. Therefore, attempting to verify the authenticity of the Support server's certificate will result in an authentication failure, and diagnostic bundle uploads will fail.

To successfully upload diagnostic bundles, you must establish trust between Cloudera Manager and the Cloudera Support server. You can accomplish this in one of the following ways:
  • Option A - Obtain the Support server's certificate and explicitly import it into your truststore.

    or

  • Option B - Use Java's default collection of public CA certificates, cacerts or jssecacerts, located at [JAVA_HOME]/jre/lib/security/, as the starting point for any truststore you create for the cluster.

The following instructions assume you have the version of keytool compatible with the version of JDK your cluster is running on.

Option A - Explicitly import the Cloudera Support server's certificate into your truststore

Use the following command to obtain the current public key/certificate information from the Cloudera Support server.

$ openssl s_client -connect cops.cloudera.com:443 | openssl x509 -text -out /path/to/cloudera-cert.pem

Import this certificate into your Cloudera Manager truststore. Substitute the paths in the following command with the paths to your truststore and the Support server's certificate.

$ keytool -import -keystore /path/to/cm/truststore.jks -file /path/to/cloudera-cert.pem
Once the truststore is ready, make sure Cloudera Manager is configured to point to this file. For instructions, see Configuring Cloudera Manager Truststore Properties.

Option B - Use the default cacerts truststore to create the Cloudera Manager truststore

You can use the default collection of public CA certificates in one of the following ways:
  • Copy the contents of cacerts to jssecacerts, and use jssecacerts as the Cloudera Manager truststore. You can modify jssecacerts to include any additional private CA certificates as needed.

    or

  • As the first step to setting up the Cloudera Manager truststore, copy the contents of cacerts to your blank truststore.jks file.
Copying the cacerts file into your truststore will also set the same default password (changeit) for your file. Use the following command to change the password for your truststore. For example: 
$ keytool -storepasswd -keystore /path/to/cm/truststore.jks 
Enter keystore password:  changeit
New keystore password:  [new-password]
Re-enter new keystore password:  [new-password]
Once the truststore is ready, use the instructions in the following section to make sure Cloudera Manager is configured to point to the right truststore file.

Configuring Cloudera Manager Truststore Properties

Once you have set up the Cloudera Manager truststore using either Option A or B, make sure Cloudera Manager is configured to use this truststore. To determine trust, Cloudera Manager uses the JVM arguments, -Djavax.net.ssl.trustStore and -Djavax.net.ssl.trustStore.password. Configure these arguments in Cloudera Manager as follows:

  1. Log into the Cloudera Manager Admin Console.
  2. Select Administration > Settings.
  3. Click the Security category.
  4. Configure the following TLS/SSL settings:
    Setting Description
    Cloudera Manager TLS/SSL Certificate Trust Store File Specify the complete filesystem path to the truststore located on the Cloudera Manager Server host in .jks format.
    Cloudera Manager TLS/SSL Certificate Trust Store Password Specify the password for the truststore file. This password is not required to access the trust store. This field can be left blank.
  5. Click Save Changes to save the settings.