Enabling HDFS Encryption Using the Wizard

1. Enable Kerberos

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

For more information about enabling Kerberos, see Enabling Kerberos Authentication for CDH.

2. Enable TLS/SSL

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

For more information about enabling TLS, see Configuring TLS/SSL Encryption for CDH Services.

3. Add a dedicated cluster for the Key Trustee Server

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

If you haven't already done so, you must create an internal repository to install Cloudera Navigator before you can set up and use Navigator Key Trustee Server. For instructions on creating internal repositories (including Cloudera Manager, CDH, and Cloudera Navigator encryption components), see Using an Internal Parcel Repository and Using an Internal Package Repository.

This step creates a new cluster in Cloudera Manager for the Key Trustee Server hosts to isolate them from other enterprise data hub (EDH) services for increased security and durability. For more information, see Data at Rest Encryption Reference Architecture.

To complete this step:

1. Click Add a dedicated cluster for the Key Trustee Server.
2. Leave Enable High Availability checked to add two hosts to the cluster. For production environments, you must enable high availability for Key Trustee Server. Failure to enable high availability can result in complete data loss in the case of catastrophic failure of a standalone Key Trustee Server. Click Continue.
3. Search for new hosts to add to the cluster, or select the Currently Managed Hosts tab to add existing hosts to the cluster. After selecting the hosts, click Continue.
4. Select the KEYTRUSTEE_SERVER parcel to install Key Trustee Server using parcels, or select None if you want to use packages. If you do not see a parcel available, click More Options and add the repository URL to the Remote Parcel Repository URLs list. After selecting a parcel or None, click Continue.

   If you selected None, click Continue again, and skip to 4. Install Key Trustee Server binary using packages or parcels.

5. After the KEYTRUSTEE_SERVER parcel is successfully downloaded, distributed, unpacked, and activated, click Continue.
6. Click Continue to complete this step and return to the main page of the wizard.

4. Install Key Trustee Server binary using packages or parcels

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

If you haven't already done so, you must create an internal repository to install Cloudera Navigator before you can set up and use Navigator Key Trustee Server. For instructions on creating internal repositories (including Cloudera Manager, CDH, and Cloudera Navigator encryption components), see Using an Internal Parcel Repository and Using an Internal Package Repository.

5. Install Parcel for Key Trustee KMS

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

If you haven't already done so, you must create an internal repository to install Cloudera Navigator before you can set up and use Navigator Key Trustee Server. For instructions on creating internal repositories (including Cloudera Manager, CDH, and Cloudera Navigator encryption components), see Using an Internal Parcel Repository and Using an Internal Package Repository.

This step installs the Key Trustee KMS parcel. If you are using packages, skip this step and see Installing Key Trustee KMS Using Packages for instructions. After installing Key Trustee KMS using packages, continue to 6. Add a Key Trustee Server Service.

6. Add a Key Trustee Server Service

Minimum Required Role: Key Administrator (also provided by Full Administrator)

This step adds the Key Trustee Server service to Cloudera Manager. To complete this step:

1. Click Add a Key Trustee Server Service.
2. Click Continue.
3. On the Customize Role Assignments for Key Trustee Server page, select the hosts for the Active Key Trustee Server and Passive Key Trustee Server roles. Make sure that the selected hosts are not used for other services (see Resource Planning for Data at Rest Encryption for more information), and click Continue.
4. The Entropy Considerations page provides commands to install the rng-tools package to increase available entropy for cryptographic operations. For more information, see Entropy Requirements. After completing these commands, click Continue.
5. The Synchronize Active and Passive Key Trustee Server Private Keys page provides instructions for generating and copying the Active Key Trustee Server private key to the Passive Key Trustee Server. Cloudera recommends following security best practices and transferring the private key using offline media, such as a removable USB drive. For convenience (for example, in a development or testing environment where maximum security is not required), you can copy the private key over the network using the provided rsync command.

   After you have synchronized the private keys, run the ktadmin init command on the Passive Key Trustee Server as described in the wizard. After the initialization is complete, check the box to indicate you have synchronized the keys and click Continue in the wizard.

6. The Setup TLS for Key Trustee Server page provides instructions on replacing the auto-generated self-signed certificate with a production certificate from a trusted Certificate Authority (CA). For more information, see Managing Key Trustee Server Certificates. Click Continue to view and modify the default certificate settings.
7. On the Review Changes page, you can view and modify the following settings:
   • Database Storage Directory (db_root)

     Default value: /var/lib/keytrustee/db

     The directory on the local filesystem where the Key Trustee Server database is stored. Modify this value to store the database in a different directory.

   • Active Key Trustee Server TLS/SSL Server Private Key File (PEM Format) (ssl.privatekey.location)

     Default value: /var/lib/keytrustee/.keytrustee/.ssl/ssl-cert-keytrustee-pk.pem

     The path to the Active Key Trustee Server TLS certificate private key. Accept the default setting to use the auto-generated private key. If you have a CA-signed certificate, change this path to the CA-signed certificate private key file. This file must be in PEM format.

   • Active Key Trustee Server TLS/SSL Server Certificate File (PEM Format) (ssl.cert.location)

     Default value: /var/lib/keytrustee/.keytrustee/.ssl/ssl-cert-keytrustee.pem

     The path to the Active Key Trustee Server TLS certificate. Accept the default setting to use the auto-generated self-signed certificate. If you have a CA-signed certificate, change this to the path to the CA-signed certificate. This file must be in PEM format.

   • Active Key Trustee Server TLS/SSL Server CA Certificate (PEM Format) (ssl.cacert.location)

     Default value: (none)

     The path to the file containing the CA certificate and any intermediate certificates (if any intermediate certificates exist, then they are required here) used to sign the Active Key Trustee Server certificate. If you have a CA-signed certificate, set this value to the path to the CA certificate or certificate chain file. This file must be in PEM format.

   • Active Key Trustee Server TLS/SSL Private Key Password (ssl.privatekey.password)

     Default value: (none)

     The password for the Active Key Trustee Server private key file. Leave this blank if the file is not password-protected.

   • Passive Key Trustee Server TLS/SSL Server Private Key File (PEM Format) (ssl.privatekey.location)

     Default value: /var/lib/keytrustee/.keytrustee/.ssl/ssl-cert-keytrustee-pk.pem

     The path to the Passive Key Trustee Server TLS certificate private key. Accept the default setting to use the auto-generated private key. If you have a CA-signed certificate, change this path to the CA-signed certificate private key file. This file must be in PEM format.

   • Passive Key Trustee Server TLS/SSL Server Certificate File (PEM Format) (ssl.cert.location)

     Default value: /var/lib/keytrustee/.keytrustee/.ssl/ssl-cert-keytrustee.pem

     The path to the Passive Key Trustee Server TLS certificate. Accept the default setting to use the auto-generated self-signed certificate. If you have a CA-signed certificate, change this to the path to the CA-signed certificate. This file must be in PEM format.

   • Passive Key Trustee Server TLS/SSL Server CA Certificate (PEM Format) (ssl.cacert.location)

     Default value: (none)

     The path to the file containing the CA certificate and any intermediate certificates (if any intermediate certificates exist, then they are required here) used to sign the Passive Key Trustee Server certificate. If you have a CA-signed certificate, set this value to the path to the CA certificate or certificate chain file. This file must be in PEM format.

   • Passive Key Trustee Server TLS/SSL Private Key Password (ssl.privatekey.password)

     Default value: (none)

     The password for the Passive Key Trustee Server private key file. Leave this blank if the file is not password-protected.

   After reviewing the settings and making any changes, click Continue.

8. After all commands complete successfully, click Continue. If the Generate Key Trustee Server Keyring appears stuck, make sure that the Key Trustee Server host has enough entropy. See Entropy Requirements for more information.
9. Click Finish to complete this step and return to the main page of the wizard.

For parcel-based Key Trustee Server releases 5.8 and higher, Cloudera Manager automatically backs up Key Trustee Server (using the ktbackup.sh script) after adding the Key Trustee Server service. It also schedules automatic backups using cron. For package-based installations, you must manually back up Key Trustee Server and configure a cron job.

Cloudera Manager configures cron to run the backup script hourly. The latest 10 backups are retained in /var/lib/keytrustee in cleartext. For information about using the backup script and configuring the cron job (including how to encrypt backups), see Backing Up Key Trustee Server and Key Trustee KMS Using the ktbackup.sh Script.

7. Add a Key Trustee KMS Service

Minimum Required Role: Key Administrator (also provided by Full Administrator)

This step adds a Key Trustee KMS service to the cluster. The Key Trustee KMS service is required to enable HDFS encryption to use Key Trustee Server for cryptographic key management. Key Trustee KMS high availability uses ZooKeeper to automatically configure load balancing. If you do not have a ZooKeeper service in your cluster, add one using the instructions in Adding a Service.

To complete this step:

1. Click Add a Key Trustee KMS Service.
2. Select an existing Key Trustee Server pair or specify an external Key Trustee Server pair. If you have an existing Key Trustee Server pair outside of Cloudera Manager control, select the External Key Trustee Server option and specify the fully-qualified domain names (FQDNs) of the Key Trustee Server pair. Click Continue.
3. Select cluster hosts for the Key Trustee KMS service. For production environments, select at least two hosts for high availability. If you proceed with only one host, you can enable high availability later. See Enabling Key Trustee KMS High Availability for more information.

   Make sure that the selected hosts are not used for other services (see Resource Planning for Data at Rest Encryption for more information), and click Continue.

4. The Entropy Considerations page provides commands to install the rng-tools package to increase available entropy for cryptographic operations. For more information, see Entropy Requirements. After completing these commands, click Continue.
5. The Setup Organization and Auth Secret page generates the necessary commands to create an organization in Key Trustee Server. An organization is required to be able to register the Key Trustee KMS with Key Trustee Server. See Managing Key Trustee Server Organizations for more information.

   Enter an organization name and click Generate Instruction. Run the displayed commands to generate an organization and obtain the auth_secret value for the organization. Enter the secret in the auth_secret field and click Continue.

6. The Setup Access Control List (ACL) page allows you to generate ACLs for the Key Trustee KMS or to provide your own ACLs. To generate the recommended ACLs, enter the username and group responsible for managing cryptographic keys and click Generate ACLs. To specify your own ACLs, select the Use Your Own kms-acls.xml File option and enter the ACLs. For more information on the KMS Access Control List, see Configuring KMS Access Control Lists (ACLs).

   After generating or specifying the ACL, click Continue.

7. The Setup TLS for Key Trustee KMS page provides high-level instructions for configuring TLS communication between the Key Trustee KMS and the Key Trustee Server, as well as between the EDH cluster and the Key Trustee KMS. See Configuring TLS/SSL for the KMS for more information.

   Click Continue.

8. The Review Changes page lists all of the settings configured in this step. Click the icon next to any setting for information about that setting. Review the settings and click Continue.
9. After the First Run commands have successfully completed, click Continue.
10. The Synchronize Private Keys and HDFS Dependency page provides instructions for copying the private key from one Key Management Server Proxy role to all other roles.
    CAUTION:

    It is very important that you perform this step. Failure to do so leaves Key Trustee KMS in a state where keys are intermittently inaccessible, depending on which Key Trustee KMS host a client interacts with, because cryptographic key material encrypted by one Key Trustee KMS host cannot be decrypted by another. If you are already running multiple Key Trustee KMS hosts with different private keys, immediately back up all Key Trustee KMS hosts, and contact Cloudera Support for assistance correcting the issue.

    To determine whether the Key Trustee KMS private keys are different, compare the MD5 hash of the private keys. On each Key Trustee KMS host, run the following command:

    $ md5sum /var/lib/kms-keytrustee/keytrustee/.keytrustee/secring.gpg

    If the outputs are different, contact Cloudera Support for assistance. Do not attempt to synchronize existing keys. If you overwrite the private key and do not have a backup, any keys encrypted by that private key are permanently inaccessible, and any data encrypted by those keys is permanently irretrievable. If you are configuring Key Trustee KMS high availability for the first time, continue synchronizing the private keys.

    Cloudera recommends following security best practices and transferring the private key using offline media, such as a removable USB drive. For convenience (for example, in a development or testing environment where maximum security is not required), you can copy the private key over the network using the provided rsync command.

    After you have synchronized the private keys, check the box to indicate you have done so and click Continue.

11. After the Key Trustee KMS service starts, click Finish to complete this step and return to the main page of the wizard.

For parcel-based Key Trustee KMS releases 5.8 and higher, Cloudera Manager automatically backs up Key Trustee KMS (using the ktbackup.sh script) after adding the Key Trustee KMS service. It does not schedule automatic backups using cron. For package-based installations, you must manually back up Key Trustee Server and configure a cron job.

The backup is stored in /var/lib/kms-keytrustee in cleartext. For more information about using the backup script and configuring the cron job (including how to encrypt backups), see Backing Up Key Trustee Server and Key Trustee KMS Using the ktbackup.sh Script.

8. Restart stale services and redeploy client configuration

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

9. Validate Data Encryption

Minimum Required Role: Key Administrator or Cluster Administrator (also provided by Full Administrator)

This step launches a tutorial with instructions on creating an encryption zone and putting data into it to verify that HDFS encryption is enabled and working.

1. Enable Kerberos

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

For more information about enabling Kerberos, see Enabling Kerberos Authentication for CDH.

2. Enable TLS/SSL

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

For more information about enabling TLS, see Configuring TLS Encryption for Cloudera Manager.

3. Install the Thales HSM Client

Before installing the Navigator HSM KMS backed by Thales HSM, you must install the Thales HSM client on the host. Attempts to install the HSM KMS service before installing the Thales HSM client will fail.

For details about how to install the Thales HSM client, refer to the Thales HSM product documentation.

4. Install Key Trustee KMS binary using parcels

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

5. Add the HSM KMS backed by Thales Service

1. Click Add Navigator HSM KMS Services backed by Thales HSM.
2. In the Thales HSM KMS Proxy field, select the hosts to which you want to assign a new or existing role. Click OK, and then click Continue.
3. To set up the ACL for the cluster, specify a comma-separated list of users and groups, and then click Generate ACLs. Click Continue.
4. Click Continue.
5. Review your selections and specify the:
   • Thales HSM Password

     Contact your HSM administrator for the Thales HSM password.

   • Keystore Password
   Then click Continue.
6. Upon notification that you have successfully added the Thales KMS Service, click Continue and Finish.

6. Restart stale services and redeploy client configuration

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

7. Validate Data Encryption

Minimum Required Role: Key Administrator or Cluster Administrator (also provided by Full Administrator)

This step launches a Validate Data Encryption tutorial with instructions describing how to create an encryption zone and place data into it to verify that HDFS encryption is enabled and working.

1. Enable Kerberos

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

For more information about enabling Kerberos, see Enabling Kerberos Authentication for CDH.

2. Enable TLS/SSL

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

For more information about enabling TLS, see Configuring TLS Encryption for Cloudera Manager.

3. Install Luna HSM Client

Before installing the Navigator HSM KMS backed by Luna HSM, you must install the Luna HSM client on the host. Attempts to install the Navigator HSM KMS backed by Luna HSM before installing the Luna HSM client will fail.

For details about how to install the Luna HSM client, refer to the Luna HSM product documentation.

4. Install Parcel for Cloudera Key Providers

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

5. Add the Navigator HSM KMS backed by SafeNet Luna HSM

1. Click Add Navigator HSM KMS backed by Safenet Luna HSM.
2. In the Luna HSM-backed KMS Proxy field, select the hosts to which you want to assign a new or existing role. Click OK, and then click Continue.
3. To set up the ACL for the cluster, specify a comma-separated list of users and groups, and then click Generate ACLs. Click Continue.
4. Click Continue.
5. Review your selections and specify the:
   • Luna HSM Password

     Contact your HSM administrator for the Luna HSM Partition password.

   • Keystore Password
   • Luna HSM Server Slot

     Identification number of the Luna HSM Server slot/device to use. If you do not know what value(s) to enter here, see the Luna product documentation for instructions on configuring your Luna HSM. Alternatively, run the /usr/safenet/lunaclient/bin/vtl verify command on the Luna HSM client host to view the slot value.

   Then click Continue.
6. Upon notification that you have successfully added the Navigator Safenet Luna KMS Service, click Continue and Finish.

6. Restart stale services and redeploy client configuration

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

7. Validate Data Encryption

Minimum Required Role: Key Administrator or Cluster Administrator (also provided by Full Administrator)

This step launches a Validate Data Encryption tutorial with instructions describing how to create an encryption zone and place data into it to verify that HDFS encryption is enabled and working.

1. Enable Kerberos

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

For more information on enabling Kerberos, see Enabling Kerberos Authentication for CDH.

2. Enable TLS/SSL

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

For more information on enabling TLS, see Configuring TLS Encryption for Cloudera Manager.

3. Add a Java KeyStore KMS Service

Minimum Required Role: Key Administrator (also provided by Full Administrator)

4. Restart stale services and redeploy client configuration

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

5. Validate Data Encryption

Minimum Required Role: Key Administrator or Cluster Administrator (also provided by Full Administrator)

This step launches a tutorial with instructions on creating an encryption zone and putting data into it to verify that HDFS encryption is enabled and working.

Limit the Number of ZooKeeper DEBUG Messages

log4j.category.org.apache.zookeeper=INFO

Limit Encryption Zone Timeouts

When creating encryption zones, there can be client timeouts due to the time it takes to fill the encrypted data encryption key (EDEK) cache. To avoid this, adjust the low watermark threshold settings as follows.

<property>
   <name>hadoop.security.kms.encrypted.key.cache.low.watermark</name>
   <value>.03</value>
</property>

<property>
   <name>hadoop.security.kms.client.encrypted.key.cache.low-watermark</name>
   <value>.02</value>
</property>

Increase KMS Client Timeout Value

Due to potential latency during installation, it is recommended that you increase the KMS client timeout value.

<property>
   <name>hadoop.security.kms.client.timeout</name>
   <value>110</value>
</property>