Integrating HDFS Encryption with Navigator Key Trustee Server

By default, HDFS Data At Rest Encryption utilizes the Hadoop Key Management Server (KMS), which uses a file-based Java KeyStore.

For increased reliability, scalability, and simplicity, Cloudera recommends instead using Cloudera Navigator Key Trustee Server (instead of the Hadoop Key Management Server) as the key store for HDFS Encryption. This topic describes how to integrate HDFS encryption on an existing CDH cluster managed by Cloudera Manager with Key Trustee Server.

If you have not installed and configured a Key Trustee Server, see Installing Cloudera Navigator Key Trustee Server.

Supported Environments

Integrating HDFS encryption with Key Trustee Server is supported using Oracle JDK 7 and 8 on the following operating systems:
  • RHEL-compatible 5, 6
  • SLES 11
  • Ubuntu 12.04, 14.04
  • Debian 7
Key Trustee Server can only be installed on RHEL-compatible OS versions 6.4, 6.5, and 6.6.

Adding Key Trustee KMS to Cloudera Manager

Install the Key Trustee KMS Binaries

Before adding the Key Trustee KMS service to Cloudera Manager, you must install the required software. See Installing Key Trustee KMS for instructions.

Add the Key Trustee KMS Service

Add the Key Trustee KMS service using the Add a Service wizard. You must have an authorization code, organization name, and the hostname of your Key Trustee Server to complete the wizard. SeeManaging Key Trustee Server Organizations for more information.

Start the Key Trustee KMS Service

If the Key Trustee KMS service does not start automatically after completing the Add a Service wizard, start it manually (Key Trustee KMS service > Actions > Start).

Configure HDFS to Use the Key Trustee KMS

Configure HDFS to be use the Key Trustee KMS service by navigating to HDFS Service > Configuration > Service-Wide and setting the KMS Service to Key Trustee KMS.

Click Save Changes and restart the HDFS service for this change to take effect. Redeploy client configuration (Home > Cluster-wide > Deploy Client Configuration).

Validating Hadoop Key Operations

Use hadoop key create to create a test key, and then use hadoop key list to retrieve the key list: 
$ hadoop key create keytrustee_test
$ hadoop key list
Use hdfs crypto to create an encryption zone. The HDFS directory used by the encryption zone must exist and be empty: 
$ hdfs crypto -listZones
$ hdfs dfs -mkdir -p /zones/testZone
$ hdfs crypto -createZone -keyName keytrustee_test -path /zones/testZone
To remove an encryption zone, delete the encrypted directory: 
$ hadoop fs -rm -r -skipTrash /zones/testZone

Key Trustee KMS High Availability

CDH 5.4.0 and higher supports Key Trustee KMS high availability. After Adding Key Trustee KMS to Cloudera Manager and Validating Hadoop Key Operations, use the following procedure to enable Key Trustee KMS high availability:
  1. Run the Add Role Instances wizard for the Key Trustee KMS service to assign a Key Management Server Proxy role to a second host.

    On the Review Changes page of the wizard, confirm the authorization code, organization name, and Key Trustee Server settings.

  2. Go to Key Trustee KMS service > Configuration and enter kms_load_balancer in the SEARCH field.
    Set the KMS Load Balancer property to the following value: 
    http://ktkms01.example.com;ktms02.example.com:16000/kms

    Replace ktkms01.example.com and ktms02.example.com with the hostnames of your Key Trustee KMS hosts. Click Save Changes and restart the Key Trustee KMS service (Key Trustee KMS service > Actions > Restart).

  3. Run the following command on the first Key Trustee KMS host: 
    rsync -zav /var/lib/kms-keytrustee/keytrustee/.keytrustee root@ktkms02.example.com:/var/lib/kms-keytrustee/keytrustee/.

    Replace ktkms02.example.com with the second Key Trustee KMS host that you are adding.

  4. Restart the Key Trustee KMS service (Key Trustee KMS service > Actions > Restart).
  5. Restart the cluster.
  6. Redeploy client configuration (Home > Cluster-wide > Deploy Client Configuration).
  7. Re-run the steps in Validating Hadoop Key Operations.

Securing the Key Trustee KMS Service

Kerberos Authentication

To enable Kerberos authentication, go to Key Trustee KMS Service > Configuration. Enter authentication in the SEARCH field. Set Authentication Type (hadoop.kms.authentication.type) to kerberos and click Save Changes.

After enabling Kerberos authentication, go to Administration > Kerberos > Credentials and click Generate Credentials.

After Kerberos credentials are generated, restart the Key Trustee KMS service in Cloudera Manager.

SSL

To enable SSL communication between the Key Trustee KMS service and Cloudera Navigator Key Trustee Server:
  1. Go to Key Trustee KMS Service > Configuration > Category > Security.
  2. Check the box labeled Enable TLS/SSL for Key Management Server Proxy.

  3. Complete the keystore and trust store configuration parameters with the location and password of the Java keystore and trust store on each host running the Key Trustee KMS service.

    The keystore must contain a valid certificate. See Creating Certificates for more information. If you are using a self-signed certificate for the Key Trustee KMS service, you must also configure HDFS clients to trust the self-signed certificate. See Configuring SSL for HDFS for instructions on configuring the truststore for HDFS clients.

    The default location for the Java truststore is $JAVA_HOME/jre/lib/security/cacerts, and the default password is changeit. If the Key Trustee Server uses a self-signed certificate, Cloudera recommends creating a new truststore. See Creating Truststores for more information.

  4. Click Save Changes.
  5. (Self-signed certificates only)
    1. SSH to each host running the Key Trustee KMS service and run the following command obtain the Key Trustee Server certificate: 
      $ sudo echo -n | openssl s_client -connect keytrustee.example.com:11371 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /tmp/keytrustee.pem
    2. Import the certificate into the truststore specified during step 3: 
      $ sudo keytool -importcert -trustcacerts  -file /tmp/keytrustee.pem -keystore /path/to/truststore -alias keytrustee.example.com
    3. Validate the import: 
      $ sudo keytool -list -v -keystore /path/to/truststore

      You should see an entry with the Key Trustee Server name you imported.

      If you have configured Key Trustee Server high availability, repeat this step for the Passive Key Trustee Server to import its certificate.

  6. (Key Trustee KMS high availability only) Go to Key Trustee KMS service > Configuration and enter kms_load_balancer in the SEARCH field.
    Update the KMS Load Balancer property to specify https instead of http: 
    https://ktkms01.example.com;ktms02.example.com:16000/kms

    Replace ktkms01.example.com and ktms02.example.com with the hostnames of your Key Trustee KMS hosts. Click Save Changes.

  7. Restart the cluster and redeploy client configuration (Home > Cluster-wide > Deploy Client Configuration).
  8. Re-run the steps in Validating Hadoop Key Operations.

Migrating Keys from a Java KeyStore to Navigator Key Trustee Server

To migrate keys from an existing Java KeyStore (JKS) (for example, if you are currently using the Java KeyStore KMS service for your key store), use the following procedure.

This procedure assumes a password-less Java KeyStore (JKS) on the same host as the new Key Trustee KMS service.
  1. Stop the Java KeyStore KMS service.
  2. Add and configure the Key Trustee KMS service, and configure HDFS to use it for its KMS Service setting. Restart the HDFS service and re-deploy client configuration (Home > Cluster-wide > Deploy Client Configuration) for this to take effect.

  3. Add the following to the Key Management Server Proxy Advanced Configuration Snippet (Safety Valve) for kms-site.xml (Key Trustee KMS Service > Configuration > Category > Advanced):

    <property>
  <name>hadoop.kms.key.provider.uri</name>
  <value>keytrustee://file@/var/lib/kms-keytrustee/keytrustee/.keytrustee/,jceks://file@/path/to/kms.keystore</value>
  <description>URI of the backing KeyProvider for the KMS.</description>
</property>
  4. Click Save Changes and restart the Key Trustee KMS service.

  5. From the host running the Key Trustee KMS service, if you have not configured Kerberos and SSL, run the following command:

    $ curl -L -d "trusteeOp=migrate" "http://kms01.example.com:16000/kms/v1/trustee/key/migrate?user.name=username&trusteeOp=migrate"

    If you have configured Kerberos and SSL, use the following command instead:

    $ curl --negotiate -u : -L -d "trusteeOp=migrate" "https://kms01.example.com:16000/kms/v1/trustee/key/migrate?user.name=username&trusteeOp=migrate" --cacert /path/to/kms/cert
  6. Monitor /var/log/kms-keytrustee/kms.log and /var/log/kms-keytrustee/kms-catalina.<date>.log to verify that the migration is successful.
  7. After you have verified that the migration is successful, remove the safety valve entry from step 3 and restart the Key Trustee KMS service.