How to Configure TLS Encryption for Cloudera Manager

The following procedure assumes that an internal certificate authority (CA) is used, and shows how to establish trust for that internal CA. If you are using a trusted public CA (such as Symantec, GeoTrust, Comodo, and others), you do not need to explicitly establish trust for the issued certificates, unless you are using an older JDK and a newer public CA. Older JDKs might not trust newer public CAs by default.

1. On the Cloudera Manager Server host, create the /opt/cloudera/security/pki directory:

   $ sudo mkdir -p /opt/cloudera/security/pki

   If you choose to use a different directory, make sure you use the same directory on all cluster hosts to simplify management and maintenance.
2. On the Cloudera Manager Server host, use the keytool utility to generate a Java keystore and certificate signing request (CSR). Replace the OU, O, L, ST, and C entries with the values for your environment. Use the same password for the -keypass and -storepass values. Cloudera Manager does not support using different passwords for the key and keystore.

   $ sudo keytool -genkeypair -alias $(hostname -f)-server -keyalg RSA -keystore \
   /opt/cloudera/security/pki/$(hostname -f)-server.jks -keysize 2048 -dname \
   "CN=$(hostname -f),OU=Engineering,O=Cloudera,L=Palo Alto,ST=California,C=US" \
   -storepass password -keypass password
   $ sudo keytool -certreq -alias $(hostname -f)-server \
   -keystore /opt/cloudera/security/pki/$(hostname -f)-server.jks \
   -file /opt/cloudera/security/pki/$(hostname -f)-server.csr -storepass password \
   -keypass password

3. Submit the CSR file (for example, cm01.example.com-server.csr) to your certificate authority to obtain a server certificate. If possible, obtain the certificate in PEM (Base64 ASCII) format. The certificate file is in PEM format if it looks like this:

   -----BEGIN CERTIFICATE-----
   MIIDAzCCAesCAQAwgY0xCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlh
   MRIwEAYDVQQHEwlQYWxvIEFsdG8xETAPBgNVBAoTCENsb3VkZXJhMRQwEgYDVQQL
   …
   tudY0C32LjGjWOg5ALliN9Oy1u2xRKGAVfapbzAZ2rchtlCZc7mtaT6BXgW8S+Db
   0HhuObn1/8TL4Ho9G+KlJB3MWik2oEbOvQt0rBidMr9qaNX86m0i7pouXZelZ5c5
   UnDPtrhW6A==
   -----END CERTIFICATE-----

   If your issued certificate is in binary (DER) format, convert it to PEM format.

4. After you receive the signed certificate, copy it to /opt/cloudera/security/pki/$(hostname -f)-server.cert.pem.
5. Copy the root and intermediate CA certificates to /opt/cloudera/security/pki/rootca.cert.pem and /opt/cloudera/security/pki/intca.cert.pem on the Cloudera Manager Server host. If you have a concatenated file containing the root CA and an intermediate CA certificate, split the file along the END CERTIFICATE/BEGIN CERTIFICATE boundary into individual files. If there are multiple intermediate CA certificates, use unique file names such as intca-1.cert.pem, intca-1.cert.pem, and so on.
6. On the Cloudera Manager Server host, copy the JDK cacerts file to jssecacerts:

   $ sudo cp $JAVA_HOME/jre/lib/security/cacerts $JAVA_HOME/jre/lib/security/jssecacerts

   If you do not have the $JAVA_HOME variable set, replace it with the path to the Oracle JDK (for example, /usr/java/jdk1.7.0_67-cloudera/).

7. Import the root CA certificate into the JDK truststore. If you do not have the $JAVA_HOME variable set, replace it with the path to the Oracle JDK.

   $ sudo keytool -importcert -alias rootca -keystore $JAVA_HOME/jre/lib/security/jssecacerts \
   -file /opt/cloudera/security/pki/rootca.cert.pem -storepass changeit

   The default password for the cacerts file is changeit. Cloudera recommends changing this password by running the command keytool -storepasswd -keystore $JAVA_HOME/jre/lib/security/cacerts.
8. Copy the jssecacerts file from the Cloudera Manager Server host to all other cluster hosts. Make sure you copy the file to the correct location ($JAVA_HOME/jre/lib/security/jssecacerts), because the Oracle JDK expects it there.
9. On the Cloudera Manager Server host, append the intermediate CA certificate to the signed server certificate, and then import it into the keystore. Make sure that you use the append operator (>>) and not the overwrite operator (>):

   $ sudo cat /opt/cloudera/security/pki/intca.cert.pem >> /opt/cloudera/security/pki/$(hostname -f)-server.cert.pem
   $ sudo keytool -importcert -alias $(hostname -f)-server \
   -file /opt/cloudera/security/pki/$(hostname -f)-server.cert.pem \
   -keystore /opt/cloudera/security/pki/$(hostname -f)-server.jks

   If you see a message like the following, enter yes to continue:

   ... is not trusted. Install reply anyway? [no]:  yes

   You must see the following response verifying that the certificate has been properly imported against its private key.

   Certificate reply was installed in keystore

   If you do not see this response, contact Cloudera Support.

Complete the following procedure on each Cloudera Manager Agent host. The provided examples continue to use an internal certificate authority (CA) to sign the agent certificates.

1. On all Cloudera Manager Agent hosts, create the /opt/cloudera/security/pki directory:

   $ sudo mkdir -p /opt/cloudera/security/pki

   If you choose to use a different directory, make sure you use the same directory on all cluster hosts to simplify management and maintenance.
2. On all Cloudera Manager Agent hosts, create a Java Keystore and private key as follows:

   $ keytool -genkeypair -alias $(hostname -f)-agent -keyalg RSA -keystore \
   /opt/cloudera/security/pki/$(hostname -f)-keystore.jks -keysize 2048 -dname \
   "CN=$(hostname -f),OU=Engineering,O=Cloudera,L=Palo Alto,ST=California,C=US" \
   -storepass password -keypass password

   Use the same password for the -keypass and -storepass values. Cloudera Manager does not support using different passwords for the key and keystore.
3. On all Cloudera Manager Agent hosts, generate the certificate signing request (CSR) and submit it to a CA. Use the keytool extended attributes to specify both serverAuth and clientAuth options:

   $ keytool -certreq -alias $(hostname -f)-agent  \
   -keystore /opt/cloudera/security/pki/$(hostname -f)-agent.jks  \
   -file /opt/cloudera/security/pki/$(hostname -f)-agent.csr  \
   -ext EKU=serverAuth,clientAuth  \
   -storepass password -keypass password

   For security purposes, many commercial CAs ignore requested extensions in a CSR. Make sure that you inform the CA that you require certificates with both server and client authentication options.

4. For each signed certificate you receive, copy it to /opt/cloudera/security/pki/$(hostname -f)-agent.cert.pem on the correct host.
5. Inspect the certificates to verify that both server and client authentication options are present:

   $ openssl x509 -in /opt/cloudera/security/pki/$(hostname -f)-agent.cert.pem -noout -text

   Look for output similar to the following:

               X509v3 Extended Key Usage:
                   TLS Web Server Authentication, TLS Web Client Authentication
   

   If the certificate does not have both TLS Web Server Authentication and TLS Web Client Authentication listed in the X509v3 Extended Key Usage section, re-submit the CSR to the CA, and request that they generate a certificate that can be used for both server and client authentication.

6. Copy the root and intermediate CA certificates to /opt/cloudera/security/pki/rootca.cert.pem and /opt/cloudera/security/pki/intca.cert.pem on each Cloudera Manager Agent host. If you have a concatenated file containing the root CA and an intermediate CA certificate, split the file along the END CERTIFICATE/BEGIN CERTIFICATE boundary into individual files. If there are multiple intermediate CA certificates, use unique file names such as intca-1.cert.pem, intca-1.cert.pem, and so on.
7. On each Cloudera Manager Agent host, append the intermediate CA certificate to the signed certificate, and then import it into the keystore. Make sure that you use the append operator (>>) and not the overwrite operator (>):

   $ sudo cat /opt/cloudera/security/pki/intca.cert.pem >> /opt/cloudera/security/pki/$(hostname -f)-agent.cert.pem
   $ sudo keytool -importcert -alias $(hostname -f)-agent \
   -file /opt/cloudera/security/pki/$(hostname -f)-agent.cert.pem \
   -keystore /opt/cloudera/security/pki/$(hostname -f)-agent.jks

   If you see a message like the following, enter yes to continue:

   ... is not trusted. Install reply anyway? [no]:  yes

   You must see the following response verifying that the certificate has been properly imported against its private key.

   Certificate reply was installed in keystore

   If you do not see this response, contact Cloudera Support.

8. On each Cloudera Manager Agent host, create symbolic links (symlink) for the certificate and keystore files:

   $ ln -s /opt/cloudera/security/pki/$(hostname -f)-agent.cert.pem /opt/cloudera/security/pki/agent.cert.pem
   $ ln -s /opt/cloudera/security/pki/$(hostname -f)-agent.jks /opt/cloudera/security/pki/agent.jks

   This allows you to use the same /etc/cloudera-scm-agent/config.ini file on all agent hosts rather than maintaining a file for each agent.

1. Log in to the Cloudera Manager Admin Console.
2. Select Administration > Settings.
3. Select the Security category.
4. Configure the following TLS settings:

   Property	Description
   Cloudera Manager TLS/SSL Server JKS Keystore File Location	The complete path to the keystore file. In this example, the path is /opt/cloudera/security/pki/cm01.example.com-server.jks. Replace cm01.example.com with the Cloudera Manager Server hostname.
   Cloudera Manager TLS/SSL Server JKS Keystore File Password	The password for the /opt/cloudera/security/jks/cm01.example.com-server.jks keystore.
   Use TLS Encryption for Admin Console	Check this box to enable TLS encryption for Cloudera Manager.

5. Click Save Changes to save the settings.

When enabling TLS for the Cloudera Manager Server admin interface, you must set the Java truststore location and password in the Cloudera Management Services configuration. Otherwise, roles such as Host Monitor and Service Monitor cannot connect to Cloudera Manager Server and will not start.

Configure the path and password for the $JAVA_HOME/jre/lib/security/jssecacerts truststore that you created earlier. Make sure that you copied this file to all cluster hosts, including the Cloudera Management Service hosts.

1. Open the Cloudera Manager Administration Console and go to the Cloudera Management Service service.
2. Click the Configuration tab.
3. Select Scope > Cloudera Management Service (Service-Wide).
4. Select Category > Security.
5. Edit the following TLS/SSL properties according to your cluster configuration.

   Property	Description
   TLS/SSL Client Truststore File Location	The path to the client truststore file used in HTTPS communication. This truststore contains certificates of trusted servers, or of Certificate Authorities trusted to identify servers. For this example, set the value to $JAVA_HOME/jre/lib/security/jssecacerts. Replace $JAVA_HOME with the path to the Oracle JDK.
   TLS/SSL Client Truststore File Password	The password for the truststore file.

6. Click Save Changes to commit the changes.

You must restart both Cloudera Manager Server and the Cloudera Management Service for TLS encryption to work. Otherwise, the Cloudera Management Services (such as Host Monitor and Service Monitor) cannot communicate with Cloudera Manager Server.

1. Restart the Cloudera Manager Server by running service cloudera-scm-server restart on the Cloudera Manager Server host.
2. After the restart completes, connect to the Cloudera Manager Admin Console using the HTTPS URL (for example: https://cm01.example.com:7183). If you used an internal CA-signed certificate, you must configure your browser to trust the certificate. Otherwise, you will see a warning in your browser any time you access the Cloudera Manager Administration Console. By default, certificates issued by public commercial CAs are trusted by most browsers, and no additional configuration is necessary if your certificate is signed by one of them.
3. Restart the Cloudera Management Service (Cloudera Management Service > Actions > Restart).

Configure the TLS properties for Cloudera Manager Agents.

Restart the Cloudera Manager Server with the following command to activate the TLS configuration settings.

$ sudo service cloudera-scm-server restart 

On each agent host, restart the Cloudera Manager agent service:

$ sudo service cloudera-scm-agent restart

In the Cloudera Manager Admin Console, go to Hosts > All Hosts. If you see successful heartbeats reported in the Last Heartbeat column after restarting the agents, TLS encryption is working properly.

On each Cloudera Manager Agent host, use the keytool utility to export the private key and certificate to a PKCS12 file, which can then be split up into individual key and certificate files using the openssl command:

1. Export the private key and certificate:

   $ keytool -importkeystore -srckeystore /opt/cloudera/security/pki/$(hostname -f)-agent.jks \
   -srcstorepass password -srckeypass password -destkeystore /opt/cloudera/security/pki/$(hostname -f)-agent.p12 \
   -deststoretype PKCS12 -srcalias $(hostname -f)-agent -deststorepass password -destkeypass password

2. Use the openssl command to export the private key into its own file:

   $ openssl pkcs12 -in /opt/cloudera/security/pki/$(hostname -f)-agent.p12 -passin pass:password -nocerts \
   -out /opt/cloudera/security/pki/$(hostname -f)-agent.key -passout pass:password

3. Create a symbolic link for the .key file:

   $ ln -s /opt/cloudera/security/pki/$(hostname -f)-agent.key /opt/cloudera/security/pki/agent.key

   This allows you to use the same /etc/cloudera-scm-agent/config.ini file on all agent hosts rather than maintaining a file for each agent.

The Cloudera Manager agent obtains the password from a text file, not from a command line parameter or environment variable. The password file allows you to use file permissions to protect the password. For example, run the following commands on each Cloudera Manager Agent host, or run them on one host and copy the file to the other hosts:

$ echo "password" > /etc/cloudera-scm-agent/agentkey.pw
$ sudo chown root:root /etc/cloudera-scm-agent/agentkey.pw
$ sudo chmod 440 /etc/cloudera-scm-agent/agentkey.pw

Replace password with the password you created in Step 1: Export the Private Key to a File.

Copy the file to all other cluster hosts. If you have modified properties such as listening_hostname or listening_ip address in config.ini, you must edit the file individually on each host.

1. Log in to the Cloudera Manager Admin Console.
2. Select Administration > Settings.
3. Click the Security category.
4. Configure the following TLS settings:

   Setting	Description
   Use TLS Authentication of Agents to Server	Select this option to enable TLS authentication of agents to the server.
   Cloudera Manager TLS/SSL Certificate Trust Store File	Specify the full filesystem path to the jssecacerts file located on the Cloudera Manager Server host. For example, /usr/java/jdk1.7.0_67-cloudera/jre/lib/security/jssecacerts.
   Cloudera Manager TLS/SSL Certificate Trust Store Password	Specify the password for the jssecacerts truststore.

5. Click Save Changes to save the settings.

1. On the Cloudera Manager server host, restart the Cloudera Manager server:

   $ sudo service cloudera-scm-server restart

2. On every agent host, restart the Cloudera Manager agent:

   $ sudo service cloudera-scm-agent restart

In the Cloudera Manager Admin Console, go to Hosts > All Hosts. If you see successful heartbeats reported in the Last Heartbeat column after restarting the agents and server, TLS certificate authentication is working properly. If not, check the agent log (/var/log/cloudera-scm-agent/cloudera-scm-agent.log) for errors.