Level 3: Configuring TLS Authentication of Agents to the Cloudera Manager Server

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

This is the highest level of TLS security supported for Cloudera Manager Server-Agent communications, and requires you to create private keys and Certificate Signing Requests (CSR) for each cluster node. A Certificate Authority (CA) can then sign the CSR, thus providing a server certificate for each host. Agents will then need to authenticate themselves to Cloudera Manager using this server certificate.

This can be completed one of two ways, depending on the approach you choose to configuring TLS on your cluster.
  • Approach A - Use OpenSSL to create private keys and request CA-signed certificates for every Agent on your cluster. Approach A is faster if you only need to enable TLS for Cloudera Manager Server-Agent communication.
  • Approach B - Create a Java truststore file that contains the Agent and CA certificates, and authenticate Agents against this truststore file. If you plan to enable TLS communication for all CDH services cluster-wide, including Java-based components, consider using Approach B.
Steps for creating self-signed certificates are not included. Self-signed certificates are not recommended for production environments.

Step 1: Configure TLS encryption

If you have not already done so, you must configure TLS encryption to use this third level of security. For instructions, see Configuring TLS Encryption Only for Cloudera Manager and Configuring TLS Encryption for Cloudera Manager.

Step 2: Configure TLS Verification of Server Trust by Agents

If you have not already done so, you must configure TLS Verification of Server Trust by Agents. For instructions, see Configuring TLS Authentication of Server to Agents.

Approach A: Using OpenSSL to Create Private Keys and Request Agent Certificates

If the Cloudera Manager Server is running Management Services or CDH components (and therefore, has a Cloudera Manager Agent installed), you do not need to re-create a private key for the Server host. Follow the steps in Private Key and Certificate Reuse Across Java Keystores and OpenSSL to reuse the host certificate. Follow steps 3-5 for all remaining cluster hosts.

Approach A Step 3: Generate the private key and certificate signing request for the Agent using OpenSSL.

Run the following command on the Agent, replacing hostname with your actual hostname. The -subj command line option allows you to provide the certificate subject as a single line. If you do not specify the certificate subject (-subj) as an argument, you will be prompted for the values of the certificate subject information. In that case, use the host FQDN that Agents will use to connect from in the subject First and Last name (CN) question prompt. Country (C) requires a 2 letter country code. The "/" is replaced with "," in the actual CSR and private key file. 
$  openssl req -subj  '/CN=hostname.sec.cloudera.com/OU=Support/O=Cloudera/L=Denver/ST=Colorado/C=US' \
-out /opt/cloudera/security/x509/hostname.csr -new -newkey rsa:2048 \ 
-keyout /opt/cloudera/security/x509/hostname.key -passout pass:password
password provides a password to protect the private key file. Keep the password in a safe place; you must provide a key password file to the Agent to complete configuration.

Approach A Step 4: Submit the certificate signing request to your CA and distribute the issued certificates.

The CSR file created (/opt/cloudera/security/x509/hostname.csr) is collected from cluster hosts for submission to the certificate authority (CA) for signing. In the example paths, you copy the issued CA-signed certificate file to /opt/cloudera/security/x509 on each cluster host. For easy management and tracking of files, name the files in the hostname.pem format, replacing hostname with the actual hostname.
The CSR can be examined with the following command: 
$  openssl req -text -noout -verify -in  /opt/cloudera/security/x509/hostname.csr
The issued certificate file can be examined with the following command: 
$ openssl x509 -in /opt/cloudera/security/x509/hostname.pem -text -noout

Approach A Step 5 (Optional): Import the OpenSSL private key and certificate into the per-host Java keystore.

Follow the steps in Private Key and Certificate Reuse Across Java Keystores and OpenSSL for this step.

Approach B: Creating a Java Keystore and Importing Signed Agent Certificates into it

If the Cloudera Manager Server is running Management Services or CDH components (and therefore, has a Cloudera Manager Agent installed), you do not need to re-create a private key for the Server host. Follow the steps in Private Key and Certificate Reuse Across Java Keystores and OpenSSL to reuse the host certificate. Follow steps 3-5 for all remaining cluster hosts.

Approach B - Step 3: Create a Java Keystore and private key for a host

Create a Java Keystore and private key files for an Agent host as follows: 
$ keytool -genkeypair -alias hostname -keyalg RSA -keystore \
/opt/cloudera/security/jks/hostname-keystore.jks -keysize 2048 -dname \
"CN=cmhost.sec.cloudera.com,OU=Support,O=Cloudera,L=Denver,ST=Colorado,C=US" \
-storepass password -keypass password
password provides a password to protect the private key file. Set -keypass to the same value as -storepass. Cloudera Manager assumes that the same password is used to access both the key and the keystore, and therefore, does not support separate values for -keypass and -storepass.

Note the password in a safe place; you must provide a key password file to the Agent to complete configuration.

Approach B - Step 4: Generate a certificate signing request and install the issued certificate into the Java Keystore

  1. Generate a certificate signing request (CSR) and submit it to your CA for a signed certificate. 
    $ keytool -certreq -alias hostname \
-keystore /opt/cloudera/security/jks/hostname-keystore.jks \ 
-file /opt/cloudera/security/x509/hostname.csr \
-storepass password -keypass password
  2. If you are using a Private CA, first import the root CA certificate followed by the intermediary/subordinate CA certificates into the Java keystore created previously. 
     $ keytool -importcert -trustcacerts -alias RootCA -keystore \
/opt/cloudera/security/jks/hostname-keystore.jks -file \ 
/opt/cloudera/security/CAcerts/RootCA.cer -storepass password

    Repeat the following for all subordinate/intermediary CA certificates presented.

    $ keytool -importcert -trustcacerts -alias SubordinateCA -keystore \
/opt/cloudera/security/jks/hostname-keystore.jks -file \ 
/opt/cloudera/security/CAcerts/SubordinateCA.cer -storepass password
  3. Copy the issued signed certificate file provided by your CA to the location from where it will be imported by the Cloudera Manager Agent and possibly Hue. 
    $ cp certificate-file.cer /opt/cloudera/security/x509/hostname.pem
  4. Import the issued certificate file into the previously created Java keystore (.jks) with the following command: 
    $ keytool -import -trustcacerts -alias <hostname> \
-keystore /opt/cloudera/security/jks/<hostname>-keystore.jks \
-file /opt/cloudera/security/x509/<hostname>.pem -storepass password

Approach B - Step 5: Export the private key from the Java keystore and convert it with OpenSSL for reuse by Agent

Follow the steps in Private Key and Certificate Reuse Across Java Keystores and OpenSSL.

Step 6: Create a File that Contains the Password for the Key

The Agent reads the password from a text file, not from the command line. The password file allows you to use file permissions to protect the password. For our example the password file was created at, /etc/cloudera-scm-agent/agentkey.pw.

Step 7: Configure the Agent with its Private Key and Certificate

  1. On the Agent host, open the /etc/cloudera-scm-agent/config.ini configuration file and edit the following properties:
    Property Description
    client_key_file Name of the client key file.
    client_keypw_file Name of the client key password file, agentkey.pw.
    client_cert_file Name of the client certificate file.
  2. Repeat these steps on every Agent host.

Step 8: Verify that steps 3-7 Were Completed for every Agent Host in Your Cluster

Make sure each Agent's private key and certificate that you import into the Cloudera Manager Server's truststore is unique.

Step 9: Create a Truststore by Importing CA and Agent Certificates

Perform this step on the Cloudera Manager Server, where the new truststore is used to authenticate Agents.

Create a new truststore file (/opt/cloudera/security/jks/truststore.jks) and import the CA root and intermediary/subordinate certificates to this truststore. The new truststore functions like a keystore, containing only certificates and no private key.

  1. Create a trusted keystore using the keytool command and import the root CA certificate to this truststore. 
    $ keytool -importcert -noprompt -keystore /opt/cloudera/security/jks/truststore.jks \
-alias root_CA -file root.crt -storepass password
  2. Import any remaining intermediary/subordinate CA certificates into the truststore. 
    $ keytool -importcert -noprompt -keystore /opt/cloudera/security/jks/truststore.jks -alias int_CA -file intermediate-CA.pem -storepass password
  3. Save the hostname.pem certificate files from all cluster hosts in a single location. The Cloudera Manager Server can now import these host certificates (hostname.pem) into the new truststore. 
    $ keytool -keystore /opt/cloudera/security/jks/truststore.jks \
-importcert -alias hostname -file hostname.pem -storepass password
    Consider creating a for loop on a list of hostnames to speed up this process. 
    $ for HOST in 'cat hostlist.txt'; do keytool -keystore /opt/cloudera/security/jks/truststore.jks \
-importcert -alias $HOST -file $HOST.pem -storepass password

Step 10: Enable Agent Authentication and Configure the Cloudera Manager Server to Use the New Truststore

  1. Log into the Cloudera Manager Admin Console.
  2. Select Administration > Settings.
  3. Click the Security category.
  4. Configure the following TLS settings:
    Setting Description
    Use TLS Authentication of Agents to Server Select this option to enable TLS authentication of Agents to the Server.
    Path to Truststore Specify the full filesystem path to the truststore located on the Cloudera Manager Server host; in the example, /opt/cloudera/security/jks/truststore.jks
    Truststore Password Specify the password for the truststore.
  5. Click Save Changes to save the settings.

Step 11: Restart the Cloudera Manager Server

$ sudo service cloudera-scm-server restart

Step 12: Restart the Cloudera Manager Agents

On every Agent host, restart the Agent:

$ sudo service cloudera-scm-agent restart

Step 13: Verify that the Server and Agents Are Communicating

In Cloudera Manager Admin Console, open the Hosts page. If the Agents heartbeat successfully, the Server and Agents are communicating. If they are not, you may see an error in the Server, such as a null CA chain error. This implies that either the truststore does not contain the Agent certificate, or the Agent is not presenting the certificate. Check all of your settings, and check the Server log to verify that TLS and Agent validation have been enabled correctly.