Level 3: Configuring the Cluster to Authenticate Agent Certificates

Required Role: Cluster Administrator or Full Administrator

Level 3 TLS configuration ensures that each Cloudera Manager Agent host presents a signed certificate to Cloudera Manager Server prior to encryption, thus preventing possible introduction of a rogue host into the cluster. The steps in this section include obtaining signed certificates from a public CA.

Prerequisites

The steps on this page assume that Level 2 TLS has already been completed:

Preliminary Task: Obtaining Certificates for the Agent

Check with the specific CA you plan to use before generating CSRs and follow their specific process.

You need to obtain certificates for each Cloudera Manager Agent host in the cluster.

Step 1: Create Directories for Security Artifacts

On each Cloudera Manager Agent host, create a directory in which to store the private key and certificates. For example: 
$ sudo mkdir -p /opt/cloudera/security/pki

Use the same directory on all cluster hosts to simplify management and maintenance.

Step 2: Generate Keys and CSRs for Cloudera Manager Agent Hosts

Generate keys and CSRs for each Cloudera Manager Agent host in the cluster using Java keytool. The command-line syntax in the steps below uses the $(hostname -f) variable to pass in the name of the host on which the command is run, to simplify the process.

On each Cloudera Manager Agent host:
  1. Create a Java keystore and private key for use by the agent host. Use the same password for the key and the keystore (Cloudera Manager requires they be the same) as shown here: 
    $ keytool -genkeypair -alias $(hostname -f)-agent -keyalg RSA -keystore \
/opt/cloudera/security/pki/$(hostname -f)-agent.jks -keysize 2048 -dname \
"CN=$(hostname -f),OU=Dept,O=Example,L=City,ST=State,C=US" \
-storepass password -keypass password
  2. Generate the CSR for certificates that can authenticate as both a client and a server, using the extended attributes serverAuth and clientAuth, as shown here: 
    $ keytool -certreq -alias $(hostname -f)-agent  \
-keystore /opt/cloudera/security/pki/$(hostname -f)-agent.jks  \
-file /opt/cloudera/security/pki/$(hostname -f)-agent.csr  \
-ext EKU=serverAuth,clientAuth  \
-storepass password -keypass password
Public CAs may sometimes ignore requested extensions in a CSR, so make sure you specify that you need certificates with both server and client authentication options.

Step 3: Submit the CSR to the CA

  1. Submit the CSR file to your certificate authority using the process and means required by the CA, for example, email or web submission.
  2. The public CA will request specific details from you, to verify that you own the domain name contained in the CSR, before they issue the certificate.
  3. When you receive the signed certificate from the CA, you can proceed with to Step 4.

Step 4: Verify the Certificate

Use OpenSSL tool to confirm that each signed certificate includes both server and client authentication options:

$ openssl x509 -in /opt/cloudera/security/pki/$(hostname -f)-agent.cert.pem -noout -text

The response should include both TLS Web Server Authentication and TLS Web Client Authentication, as shown below:

  X509v3 Extended Key Usage:
    TLS Web Server Authentication, TLS Web Client Authentication
If either value is missing from the certificate's X509v3 Extended Key Usage section:
  1. Check your CSR and make sure it was created using both extended attribute options,
  2. Re-submit the CSR to the CA and re-iterate that the signed certificate must include both options, so that it can authenticate as a server and a client.

Assuming you have properly constructed certificates, you can now distributed the certificates to the Cloudera Manager Agent host comprising the cluster.

Concatenated Certificates and Multiple Intermediate Certificates

If the CA provided you with the root CA and intermediate file concatenated into a single file, split the file along the END CERTIFICATE/BEGIN CERTIFICATE boundary into individual files.

If the CA provided you with multiple intermediate CA certificates, use unique file names for each, such as intca-1.cert.pem, intca-2.cert.pem, and so on.

Use these certificates as needed in Step 5.

Step 5: Distribute the Certificates to the Cloudera Manager Agent Hosts

On each Cloudera Manager Agent host:

  1. Copy each signed certificate to the appropriate host in the path shown below. Add a suffix -agent.cert to each for easy identification (as to functionality) as shown here: 
    /opt/cloudera/security/pki/$(hostname -f)-agent.cert.pem
  2. Copy any root and intermediate CA certificates to the following paths: 
    /opt/cloudera/security/pki/rootca.cert.pem
/opt/cloudera/security/pki/intca.cert.pem
  3. Append the intermediate CA certificate to the signed certificate. Note the append operator (>>) in the command below, not the overwrite operator (>): 
    $ sudo cat /opt/cloudera/security/pki/intca.cert.pem >> /opt/cloudera/security/pki/$(hostname -f)-agent.cert.pem
  4. Import the signed certificate into the keystore: 
    $ sudo keytool -importcert -alias $(hostname -f)-agent \
-file /opt/cloudera/security/pki/$(hostname -f)-agent.cert.pem \
-keystore /opt/cloudera/security/pki/$(hostname -f)-agent.jks
    The command should return a message such as that shown below. Enter yes at the prompt to continue: 
    ... is not trusted. Install reply anyway? [no]:  yes
    You should see a message confirming successful import of the signed certificate into the keystore, certifying the private key: 
    Certificate reply was installed in keystore

Assuming the certificate imported successfully, you can continue.

Configuring the Cloudera Manager Agent Hosts for Certificate Authentication

The steps below assume that the Preliminary Tasks (obtaining and distributing the signed certificates) are complete.

Step 7: Export the Private Key to a File

Repeat these steps each Cloudera Manager Agent host system.

On each Cloudera Manager Agent host, use the keytool utility to export the private key and certificate to a PKCS12 file, which can then be split up into individual key and certificate files using the openssl command:

  1. Create a PKCS12 version of the Java keystore: 
    $ keytool -importkeystore -srckeystore /opt/cloudera/security/pki/$(hostname -f)-agent.jks \
-srcstorepass password -srckeypass password \
-destkeystore /opt/cloudera/security/pki/$(hostname -f)-agent.p12 \
-deststoretype PKCS12 -srcalias $(hostname -f)-agent -deststorepass \
password -destkeypass password
  2. Use the openssl command to export the private key into its own file: 
    $ openssl pkcs12 -in /opt/cloudera/security/pki/$(hostname -f)-agent.p12 \
-passin pass:password -nocerts -out \
/opt/cloudera/security/pki/$(hostname -f)-agent.key -passout pass:password
  3. Create a symbolic link for the .key filename: 
    $ ln -s /opt/cloudera/security/pki/$(hostname -f)-agent.key /opt/cloudera/security/pki/agent.key

    With the symlink pointing to agent.key on each Cloudera Manager Agent host in the cluster, the same /etc/cloudera-scm-agent/config.ini file can also be used for all agent hosts rather than maintaining individual files. (The configuration changes are made in Step 8, below.)

Step 8: Create a Password File

The Cloudera Manager agent obtains the password from a text file, not from a command line parameter or environment variable. The password file allows you to use file permissions to protect the password. For example, run the following commands on each Cloudera Manager Agent host, or run them on one host and copy the file to the other hosts:
  1. Use a text editor to create a file called agentkey.pw that contains the password. Save the file in the /etc/cloudera-scm-agent directory.
  2. Change ownership of the file to root: 
    $ sudo chown root:root /etc/cloudera-scm-agent/agentkey.pw
  3. Change the permissions of the file: 
    $ sudo chmod 440 /etc/cloudera-scm-agent/agentkey.pw
  • The password used in the echo command is the private key that was created when TLS was initially configured for the cluster (see Export the Private Key to a File).
  • The path to the agentkey.pw should match the path specified by the client_keypw_file property in the Agent's config.ini file.

Step 9: Configure the Agent to Use Private Keys and Certificates

Edit the configuration file on one of the Cloudera Manager Agent hosts in the cluster, as follows:
  1. Use a text editor to open the configuration file, located in this path: 
    /etc/cloudera-scm-agent/config.ini
  2. In the [Security] section in the file (just above the [Hadoop] section, find the PEM certificates section details, specifically:
    1. client_key_file, which identifies the path to the private key file.
    2. client_keypw_file, which identifies the path to the private key password file.
    3. client_cert_file, which identifies the path to the client certificate file.
  3. Remove the hash mark (#) at the start of the line and enter the values for your system, as shown below (in bold) using the settings created so far: 
    [security]
...
# PEM file containing client private key
client_key_file=/opt/cloudera/security/pki/agent.key
...
# If client_keypw_cmd isn't specified, instead a text file contining
# the client private key password can be used.
client_keypw_file=/etc/cloudera-scm-agent/agentkey.pw

# PEM file containing client certificate.
client_cert_file=/opt/cloudera/security/pki/agent.cert.pem
...

Copy the file to all other cluster hosts. If you have modified properties such as listening_hostname or listening_ip address in config.ini, you must edit the file individually on each host.

Step 10: Enable Agent Certificate Authentication

  1. Log in to the Cloudera Manager Admin Console.
  2. Select Administration > Settings.
  3. Click the Security category.
  4. Configure the following TLS settings:
    Setting Description
    Use TLS Authentication of Agents to Server Click the box to enable TLS authentication for Cloudera Manager Agent hosts. By default, the box is unchecked.
    Cloudera Manager TLS/SSL Certificate Trust Store File Enter the full filesystem path to the jssecacerts file (created in Step 2: Create the Java Truststore) on the Cloudera Manager Server host. For example: /usr/java/jdk1.7.0_67-cloudera/jre/lib/security/jssecacerts
    Cloudera Manager TLS/SSL Certificate Trust Store Password Specify the password for the jssecacerts truststore.
  5. Click Save Changes to save the settings.

Step 11: Restart Cloudera Manager Server and Agents

  1. On the Cloudera Manager server host, restart the Cloudera Manager server: 
    $ sudo service cloudera-scm-server restart
  2. On every agent host, restart the Cloudera Manager agent: 
    $ sudo service cloudera-scm-agent restart

Step 12: Verify Cloudera Manager Server and Agent Communications

In the Cloudera Manager Admin Console, go to Hosts > All Hosts. If you see successful heartbeats reported in the Last Heartbeat column after restarting the agents and server, TLS certificate authentication is working properly. If not, check the agent log (/var/log/cloudera-scm-agent/cloudera-scm-agent.log) for errors. See TLS Level 2, Step 4 for details about accessing the log.