Creating Java Keystores and Truststores

Typically, a keystore is used in one of two distinct ways:
  • The keystore contains private keys and certificates used by TLS/SSL servers to authenticate themselves to TLS/SSL clients. By convention, such files are referred to as keystores.
  • When used as a truststore, the file contains certificates of trusted TLS/SSL servers, or of Certificate Authorities trusted to identify servers. There are no private keys in the truststore.
While all TLS/SSL clients must have access to a truststore, it is not always necessary to create and deploy truststores across a cluster. The standard JDK distribution includes a default truststore which is pre-provisioned with the root certificates of a number of well-known Certificate Authorities. If you do not provide a custom truststore, the Hadoop daemons load this default truststore. Therefore, if you are using certificates issued by a CA in the default truststore, you do not need to provide custom truststores. However, you must consider the following before you decide to use the default truststore:
  • If you choose to use the default truststore, it is your responsibility to maintain it. You may need to remove the certificates of CAs you do not deem trustworthy, or add or update the certificates of CAs you trust. Use the keytool utility to perform these actions.

Continue reading:

Security Considerations for Keystores and Truststores

Because keystores contain private keys, while truststores do not, the security requirements for keystores are more stringent. In particular:

  • Hadoop TLS/SSL requires that truststores and the truststore password be stored, in plaintext, in a configuration file that is readable by all.
  • Keystore and key passwords are stored, in plaintext, in a file that is readable only by members of the appropriate group.

These considerations should inform your choice of which keys and certificates to store in the keystores and truststores you will deploy across your cluster.

  • Keystores should contain a minimal set of keys and certificates. A reasonable strategy would be to create a unique keystore for each host, which would contain only the keys and certificates needed by the Hadoop TLS/SSL services running on the host. In most cases, the keystore would contain a single key/certificate entry.

    Modifying Keystores: CDH services and processes must be restarted in case changes are made to a keystore. However, this is relatively rare since keystores do not need to be updated when hosts are added or deleted from a cluster.

  • Because truststores do not contain sensitive information, it is reasonable to create a single truststore for an entire cluster. On a production cluster, such a truststore would often contain a single CA certificate (or certificate chain), since you would typically choose to have all certificates issued by a single CA.
  • Since truststore passwords are stored in the clear in files readable by all, doing so would compromise the security of the private keys in the keystore.

Creating Keystores

Once you have settled on a storage plan for your keys and certificates, you can use keytool to create or update the necessary keystores and truststores. To create a new keystore with a certificate see Creating Certificates.

In many cases, you will already have created the set of keystores that you need. If you have followed the approach of creating a separate keystore for each private key and certificate, and wish to maintain this arrangement when deploying the keystores, no additional steps are required to prepare the keystores for deployment. If you wish to reorganize your keys and certificates into a different set of keystores, you can use keytool -importkeystore to transfer entries from one keystore to another.

Creating Truststores

The steps involved in preparing the truststores to be used in your deployment depend on whether you have decided to use the default Java truststore, or to create custom truststores:
  • If you are using the default truststore, you may need to add CA certificates (or certificate chains) to the truststore, or delete them from the truststore.
  • If you are creating custom truststores, you will need to build the truststores by importing trusted certificates into new truststores. The trusted certificates can be CA certificates (typically downloaded from the CA's website), or self-signed certificates that you have created. Note that Cloudera strongly recommends against using self-signed certificates in production.

As shown in the examples below, when creating a truststore you must select a password. All truststore passwords for a given service must be the same. In practice, this restriction rarely comes into play, since it is only relevant when you wish to create distinct custom truststores for each host.

The following sections provide examples of the steps required for several common scenarios:

Example 1: Adding a CA Certificate to the alternative Default Truststore

In this example, we assume that you have chosen to use the default Java truststore, but have obtained a certificate from a CA not included in the truststore. (This situation can also arise if the CA that issued your certificate has an entry in the default truststore, but the particular certificate product you purchased requires an alternate CA certificate chain.)

  1. Locate the default truststore on your system. The default truststore is located in the $JAVA_HOME/jre/lib/security/cacerts file. This contains the default CA information shipped with the JDK. Create an alternate default file called jssecacerts in the same location as the cacerts file. You can now safely append CA certificates for any private or public CAs not present in the default cacerts file, while keeping the original file intact.

    The alternate file will always be read unless the javax.net.ssl.trustStore flag is set in the arguments for the startup of the java process.

    For our example, we will be following this recommendation by copying the default cacerts file into the new jssecacerts file. 
    $ cp $JAVA_HOME/jre/lib/security/cacerts \
  $JAVA_HOME/jre/lib/security/jssecacerts
    If you use a copy of the cacerts file, remember the default keystore password is changeit.
  2. Import the CA certificate into the default truststore. Assuming that the file CA-root.cer contains the CA’s certificate, which you have previously downloaded from the CA’s web site, the following command imports this certificate into the alternative default truststore. 
    $ keytool -importcert -file CA-root.cer -alias CAcert \
-keystore /usr/java/default/jre/lib/security/jssecacerts \
-storepass changeit
    When you give this command, you will be prompted to confirm that you trust the certificate. Be sure to verify that the certificate is genuine before importing it.

Example 2: Creating a Custom Truststore Containing a Single CA Certificate Chain

In this example, we demonstrate how to use keytool to create a custom truststore. We assume all certificates were issued by a single CA, so a truststore containing the certificate chain for that CA will serve for all hosts in the cluster.

Our example certificate chain consists of a root certificate and a single intermediate certificate. We assume that you have downloaded these and saved them in the files CA-root.cer and CA-intermediate.cer (respectively). The steps below show the commands needed to build a custom truststore containing the root and intermediate certificates.

  1. Import the root certificate and create the truststore: 
    $ keytool -importcert -keystore custom.truststore -alias CA-cert \
-storepass trustchangeme -file CA-root.cer
    You will be prompted to confirm that the root certificate is trustworthy. Be sure to verify that the certificate is genuine before you import it.
  2. Import the intermediate certificate into the truststore created in Step 1: 
    $ keytool -importcert -keystore custom.truststore \
-alias CA-intermediate -storepass trustchangeme \
-file CA-intermediate.cer

Example 3: Creating a Custom Truststore Containing Self-Signed Test Certificates

This example is particularly relevant when setting up a test cluster. We assume that you have generated a set of self-signed test certificates for the hosts in the cluster, and wish to create a single truststore that can be deployed on all hosts. Because the certificates are self-signed, we cannot simply construct a truststore containing a single certificate chain, as in the previous example. When a client receives a self-signed certificate from a server during the TLS/SSL handshake, it must be able to find the server’s certificate in the truststore, since no other signing certificate exists to establish trust. Therefore, the truststore must contain all the test certificates.

We assume that the test certificates reside in keystores named node1.keystore … node100.keystore, which were created following the steps described in Creating Self-Signed Test Certificates.

  1. Export the test certificate for node1.example.com: 
    $ keytool -exportcert -keystore node1.keystore -alias node1 \
-storepass changeme -file node1.cer
  2. Import the test certificate into the custom truststore: 
    keytool -importcert -keystore custom.truststore -alias node1 \
-storepass trustchangeme -file node1.cer -noprompt
    Here we specify the -noprompt option to suppress the prompt asking you to confirm that the certificate is trustworthy. Since you created the certificate yourself, this confirmation is unnecessary.
  3. Repeat Steps 1 and 2 for node2.keystore … node100.keystore.