Level 3: Configuring TLS Authentication of Agents to the Cloudera Manager Server

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

This is the highest level of TLS security supported for Cloudera Manager Server-Agent communications, and requires you to create private keys and Certificate Signing Requests (CSR) for each cluster node. A Certificate Authority (CA) can then sign the CSR, thus providing a server certificate for each host. Agents will then need to authenticate themselves to Cloudera Manager using this server certificate.

This can be completed one of two ways, depending on the approach you choose to configuring TLS on your cluster.
  • Approach A - Use OpenSSL to create private keys and request CA-signed certificates for every Agent on your cluster. Approach A is faster if you only need to enabling TLS for Cloudera Manager Server-Agent communication.
  • Approach B - Create a Java truststore file that contains the Agent and CA certificates, and authenticate Agents against this truststore file. If you plan to enable TLS communication for all CDH services cluster-wide, including Java-based components, consider using Approach B.
Self-signed certificates have not documented since Cloudera does not recommend using them within enterprise production environments.

Step 1: Configure TLS encryption

If you have not already done so, you must configure TLS encryption to use this third level of security. For instructions, see Configuring TLS Encryption Only for Cloudera Manager and Configuring TLS Encryption for Cloudera Manager.

Step 2: Configure TLS Verification of Server Trust by Agents

If you have not already done so, you must configure TLS Verification of Server Trust by Agents. For instructions, see Configuring TLS Authentication of Server to Agents.

Approach A: Using OpenSSL to Create Private Keys and Request Agent Certificates

If the Cloudera Manager Server is running Management Services or CDH components (and therefore, has a Cloudera Manager Agent installed), you do not need to re-create a private key for the Server host. You can follow the steps indicated in Private Key and Certificate Reuse Across Java Keystores and OpenSSL to reuse the host certificate. Follow steps 3-5 for all remaining cluster hosts.

Approach A - Step 3. Generate the private key and certificate signing request for the Agent using OpenSSL.

Run the following command on the Agent, replacing hostname with your actual hostname. The -subj command line option allows you to provide the certificate subject as a single line. If not specified, you will be prompted for the values of the certificate subject information. In that case, use the host FQDN that Agents will use to connect from in the subject First and Last name (CN) question prompt. Note that Country (C) requires a 2 letter country code. The "/" is replaced with "," in the actual CSR and private key file. 
$  openssl req -subj  '/CN=hostname.sec.cloudera.com/OU=Support/O=Cloudera/L=Denver/ST=Colorado/C=US' \
-out /opt/cloudera/security/x509/hostname.csr -new -newkey rsa:2048 \ 
-keyout /opt/cloudera/security/x509/hostname.key -passout pass:password
password in the command above, provides a password to protect the private key file. Note the password in a safe place since you will be required to provide a key password file to the Agent to complete configuration.

Approach A - Step 4: Submit the certificate signing request to your CA and distribute the issued certificates.

The CSR file created (/opt/cloudera/security/x509/hostname.csr) will be gathered from cluster hosts for submission to the Certificate Authority (CA) for signing. According to our example paths, you need to copy the issued CA-signed certificate file to /opt/cloudera/security/x509 on each cluster host. For easy management/tracking of files, you can name the files in the hostname.pem format, replacing hostname with the actual host’s name.
The CSR can be examined with the following command 
$  openssl req -text -noout -verify -in  /opt/cloudera/security/x509/hostname.csr
The issued certificate file can be examined with the following command. 
$ openssl x509 -in /opt/cloudera/security/x509/hostname.pem -text -noout

Approach A - (Optional) Step 5: Import the OpenSSL private key and certificate into the per-host Java keystore.

You can follow the steps indicated in the reference section, Private Key and Certificate Reuse Across Java Keystores and OpenSSL for this step.

Approach B: Creating a Java Keystore and Importing Signed Agent Certificates into it

If the Cloudera Manager Server is running Management Services or CDH components (and therefore, has a Cloudera Manager Agent installed), you do not need to re-create a private key for the Server host. You can follow the steps indicated in Private Key and Certificate Reuse Across Java Keystores and OpenSSL to reuse the host certificate. Follow steps 3-5 for all remaining cluster hosts.

Approach B - Step 3: Create a Java Keystore and private key for a host

Create Java Keystore and private key files for an Agent host as follows: 
$ keytool -genkey -alias hostname -keyalg RSA -keystore \
/opt/cloudera/security/jks/hostname-keystore.jks -keysize 2048 -dname \
"CN=cmhost.sec.cloudera.com,OU=Support,O=Cloudera,L=Denver,ST=Colorado,C=US" \
-storepass password -keypass password
password in the command above, provides a password to protect the private key file. Note the password in a safe place since you will be required to provide a key password file to the Agent to complete configuration.

Approach B - Step 4: Generate a certificate signing request and install the issued certificate into the Java Keystore

  1. Generate a Certificate Signing Request (CSR) and submit it to your CA for a signed certificate. 
    $ keytool -certreq -alias hostname \
-keystore /opt/cloudera/security/jks/hostname-keystore.jks \ 
-file /opt/cloudera/security/x509/hostname.csr \
-storepass password -keypass password
  2. If you are using a Private CA, make sure you first import the root CA certificate followed by the intermediary/subordinate CA certificates into the Java keystore created above. 
     $ keytool -import -trustcacerts -alias RootCA -keystore \
/opt/cloudera/security/jks/hostname-keystore.jks -file \ 
/opt/cloudera/security/CAcerts/RootCA.cer -storepass password

    Repeat the following for as many subordinate/intermediary CA certificates that are presented.

    $ keytool -import -trustcacerts -alias SubordinateCA -keystore \
/opt/cloudera/security/jks/hostname-keystore.jks -file \ 
/opt/cloudera/security/CAcerts/SubordinateCA.cer -storepass password
  3. Copy the issued signed certificate file provided by your CA to the location from where it will be imported by the Cloudera Manager Agent and possibly Hue. 
    $ cp certificate-file.cer /opt/cloudera/security/x509/hostname.pem
  4. Import that issued certificate file into the previously created Java keystore (.jks) with the following keytool command 
    $ keytool -import -trustcacerts -alias <hostname> \
-keystore /opt/cloudera/security/jks/<hostname>-keystore.jks \
-file /opt/cloudera/security/x509/<hostname>.pem -storepass password

Approach B - Step 5: Export private key from Java keystore and convert with OpenSSL for reuse by Agent.

You can follow the steps indicated in the reference section, Private Key and Certificate Reuse Across Java Keystores and OpenSSL for this step.

Step 6: Create a File that Contains the Password for the Key

The Agent reads the password from a text file, and not from command line. The password file allows you to use file permissions to protect the password. For our example the password file was created at, /etc/cloudera-scm-agent/agentkey.pw.

Step 7: Configure the Agent with its Private Key and Certificate

  1. On the Agent host, open the /etc/cloudera-scm-agent/config.ini configuration file:
  2. Edit the following properties in the /etc/cloudera-scm-agent/config.ini configuration file.
    Property Description
    client_key_file Name of the client key file.
    client_keypw_file Name of the client key password file, agentkey.pw
    client_cert_file Name of the client certificate file
  3. Repeat these steps on every Agent host.

Step 8: Verify that steps 3-7 Were Completed for every Agent Host in Your Cluster

Step 9: Create a Truststore by Importing CA and Agent Certificates

Perform this step on the Cloudera Manager server, where the new truststore will be used to authenticate Agents.

Create a new truststore file (/opt/cloudera/security/jks/truststore.jks) and import the CA root and intermediary/subordinate certificates to this truststore. The new truststore is essentially like any another keystore, containing only certificates and no private key.
  1. Create a trusted keystore using the keytool command and import the root CA certificate to this truststore. 
    $ keytool -import -noprompt -keystore /opt/cloudera/security/jks/truststore.jks \
-alias root_CA -file root.crt -storepass password
  2. Import any remaining intermediary/subordinate CA certificates into the truststore. 
    $ keytool -import -noprompt -keystore /opt/cloudera/security/jks/truststore.jks -alias int_CA -file intermediate-CA.pem -storepass password
  3. Finally, gather all the hostname.pem certificate files from all the cluster hosts to a location that can be accessed by the Cloudera Manager Server. You can now import all the host certificates (hostname.pem), into the new truststore. 
    $ keytool -keystore /opt/cloudera/security/jks/truststore.jks \
-import -alias hostname -file hostname.pem -storepass password
    Consider creating a for loop on a list of hostnames to speed up this process. 
    $ for HOST in 'cat hostlist.txt'; do keytool -keystore /opt/cloudera/security/jks/truststore.jks \
-import -alias $HOST -file $HOST.pem -storepass password

Step 10: Enable Agent Authentication and Configure the Cloudera Manager Server to Use the New Truststore

  1. Log into the Cloudera Manager Admin Console.
  2. Select Administration > Settings.
  3. Click the Security category.
  4. Configure the following TLS settings:
    Setting Description
    Use TLS Authentication of Agents to Server Select this option to enable TLS Authentication of Agents to the Server.
    Path to Truststore Specify the full filesystem path to the truststore located on the Cloudera Manager Server host. For our example, /opt/cloudera/security/jks/truststore.jks
    Truststore Password Specify the password for the truststore.
  5. Click Save Changes to save the settings.

Step 12: Restart the Cloudera Manager Server

$ sudo service cloudera-scm-server restart

Step 13: Restart the Cloudera Manager Agents

On every Agent host, restart the Agent:

$ sudo service cloudera-scm-agent restart

Step 14: Verify that the Server and Agents Are Communicating

In Cloudera Manager Admin Console, open the Hosts page. If the Agents heartbeat successfully, the Server and Agents are communicating. If they are not, you may get an error in the Server, such as a null CA chain error. This implies either the truststore doesn't contain the Agent certificate or the Agent isn't presenting the certificate. Double check all of your settings. Check the Server's log to verify whether TLS and Agent validation have been enabled correctly.