Enabling HDFS Encryption Using the Wizard

To accommodate the security best practice of separation of duties, enabling HDFS encryption using the wizard requires different Cloudera Manager user roles for different steps.

Launch the Set up HDFS Data At Rest Encryption wizard in one of the following ways:
  • Cluster > Set up HDFS Data At Rest Encryption

    Minimum Required Role: Key Administrator or Cluster Administrator (also provided by Full Administrator) This feature is not available when using Cloudera Manager to manage Data Hub clusters.

  • Administration > Security > Set up HDFS Data At Rest Encryption

    Minimum Required Role: Key Administrator or Cluster Administrator (also provided by Full Administrator) This feature is not available when using Cloudera Manager to manage Data Hub clusters.

  • HDFS service > Actions > Set up HDFS Data At Rest Encryption

    Minimum Required Role: Cluster Administrator (also provided by Full Administrator) This feature is not available when using Cloudera Manager to manage Data Hub clusters.

On the first page of the wizard, select the root of trust for encryption keys:
  • Cloudera Navigator Key Trustee Server
  • Navigator HSM KMS backed by Thales HSM
  • Navigator HSM KMS backed by Luna HSM
  • A file-based password-protected Java KeyStore
Cloudera strongly recommends using Cloudera Navigator Key Trustee Server as the root of trust for production environments. The file-based Java KeyStore root of trust is insufficient to provide the security, scalability, and manageability required by most production systems. More specifically, the Java KeyStore KMS does not provide:
  • Scalability, so you are limited to only one KMS, which can result in bottlenecks
  • High Availability (HA)
  • Recoverability, so if you lose the node where the Java KeyStore is stored, then you can lose access to all the encrypted data

Ultimately, the Java KeyStore does not satisfy the stringent security requirements of most organizations for handling master encryption keys.

Choosing a root of trust displays a list of steps required to enable HDFS encryption using that root of trust. Each step can be completed independently. The Status column indicates whether the step has been completed, and the Notes column provides additional context for the step. If your Cloudera Manager user account does not have sufficient privileges to complete a step, the Notes column indicates the required privileges.

Available steps contain links to wizards or documentation required to complete the step. If a step is unavailable due to insufficient privileges or a prerequisite step being incomplete, no links are present and the Notes column indicates the reason the step is unavailable.

Continue to the section for your selected root of trust for further instructions:

Enabling HDFS Encryption Using Navigator Key Trustee Server

Enabling HDFS encryption using Key Trustee Server as the key store involves multiple components. For an overview of the components involved in encrypting data at rest, see “Encrypting Data at Rest”. For guidelines on deploying the Navigator Key Trustee Server in production environments, “Data at Rest Encryption Requirements”.

Before continuing, make sure the Cloudera Manager server host has access to the internal repository hosting the Key Trustee Server software.

After selecting Cloudera Navigator Key Trustee Server as the root of trust, the following steps are displayed:

1. Enable Kerberos

Minimum Required Role: Cluster Administrator (also provided by Full Administrator) This feature is not available when using Cloudera Manager to manage Data Hub clusters.

2. Enable TLS/SSL

Minimum Required Role: Cluster Administrator (also provided by Full Administrator) This feature is not available when using Cloudera Manager to manage Data Hub clusters.

3. Add a dedicated cluster for the Key Trustee Server

Minimum Required Role: Cluster Administrator (also provided by Full Administrator) This feature is not available when using Cloudera Manager to manage Data Hub clusters.

If you haven't already done so, you must create an internal repository to install Cloudera Navigator before you can set up and use Navigator Key Trustee Server.

This step creates a new cluster in Cloudera Manager for the Key Trustee Server hosts to isolate them from other enterprise data hub (EDH) services for increased security and durability. For more information, see “Data at Rest Encryption Reference Architecture”.

To complete this step:

  1. Click Add a dedicated cluster for the Key Trustee Server.
  2. Leave Enable High Availability checked to add two hosts to the cluster. For production environments, you must enable high availability for Key Trustee Server. Failure to enable high availability can result in complete data loss in the case of catastrophic failure of a standalone Key Trustee Server. Click Continue.
  3. Search for new hosts to add to the cluster, or select the Currently Managed Hosts tab to add existing hosts to the cluster. After selecting the hosts, click Continue.
  4. Select the KEYTRUSTEE_SERVER parcel to install Key Trustee Server using parcels, or select None if you want to use packages. If you do not see a parcel available, click More Options and add the repository URL to the Remote Parcel Repository URLs list. After selecting a parcel or None, click Continue.

    If you selected None, click Continue again, and skip to 4. Install Key Trustee Server binary using packages or parcels.

  5. After the KEYTRUSTEE_SERVER parcel is successfully downloaded, distributed, unpacked, and activated, click Continue.
  6. Click Continue to complete this step and return to the main page of the wizard.

4. Install Key Trustee Server binary using packages or parcels

Minimum Required Role: Cluster Administrator (also provided by Full Administrator) This feature is not available when using Cloudera Manager to manage Data Hub clusters.

If you haven't already done so, you must create an internal repository to install Cloudera Navigator before you can set up and use Navigator Key Trustee Server. For instructions on creating internal repositories (including Cloudera Manager, CDP, and Cloudera Navigator encryption components), see Configuring Local Package and Parcel Repositories.

This step is completed automatically during 3. Add a dedicated cluster for the Key Trustee Server if you are using parcels. If the step is incomplete for any reason (such as the wizard being interrupted or a failure installing the parcel), complete it manually:
  1. Click Install Key Trustee Server binary using packages or parcels.
  2. Select the KEYTRUSTEE_SERVER parcel to install Key Trustee Server, or select None if you need to install Key Trustee Server manually using packages. If you do not see a parcel available, click More Options and add the repository URL to the Remote Parcel Repository URLs list. After selecting a parcel, click Continue.
  3. After the KEYTRUSTEE_SERVER parcel is successfully downloaded, distributed, unpacked, and activated, click Finish to complete this step and return to the main page of the wizard.

6. Add a Key Trustee Server Service

Minimum Required Role: Key Administrator (also provided by Full Administrator)

This step adds the Key Trustee Server service to Cloudera Manager. To complete this step:

  1. Click Add a Key Trustee Server Service.
  2. Click Continue.
  3. On the Customize Role Assignments for Key Trustee Server page, select the hosts for the Active Key Trustee Server and Passive Key Trustee Server roles. Make sure that the selected hosts are not used for other services (see “Resource Planning for Data at Rest Encryption” for more information), and click Continue.
  4. The Entropy Considerations page provides commands to install the rng-tools package to increase available entropy for cryptographic operations. For more information, see “Data at Rest Encryption Requirements”. After completing these commands, click Continue.
  5. The Synchronize Active and Passive Key Trustee Server Private Keys page provides instructions for generating and copying the Active Key Trustee Server private key to the Passive Key Trustee Server. Cloudera recommends following security best practices and transferring the private key using offline media, such as a removable USB drive. For convenience (for example, in a development or testing environment where maximum security is not required), you can copy the private key over the network using the provided rsync command.

    After you have synchronized the private keys, run the ktadmin init command on the Passive Key Trustee Server as described in the wizard. After the initialization is complete, check the box to indicate you have synchronized the keys and click Continue in the wizard.

  6. The Setup TLS for Key Trustee Server page provides instructions on replacing the auto-generated self-signed certificate with a production certificate from a trusted Certificate Authority (CA). For more information, see “Managing Key Trustee Server Certificates”. Click Continue to view and modify the default certificate settings.
  7. On the Review Changes page, you can view and modify the following settings:
    • Database Storage Directory (db_root)

      Default value: /var/lib/keytrustee/db

      The directory on the local filesystem where the Key Trustee Server database is stored. Modify this value to store the database in a different directory.

    • Active Key Trustee Server TLS/SSL Server Private Key File (PEM Format) (ssl.privatekey.location)

      Default value: /var/lib/keytrustee/.keytrustee/.ssl/ssl-cert-keytrustee-pk.pem

      The path to the Active Key Trustee Server TLS certificate private key. Accept the default setting to use the auto-generated private key. If you have a CA-signed certificate, change this path to the CA-signed certificate private key file. This file must be in PEM format.

    • Active Key Trustee Server TLS/SSL Server Certificate File (PEM Format) (ssl.cert.location)

      Default value: /var/lib/keytrustee/.keytrustee/.ssl/ssl-cert-keytrustee.pem

      The path to the Active Key Trustee Server TLS certificate. Accept the default setting to use the auto-generated self-signed certificate. If you have a CA-signed certificate, change this to the path to the CA-signed certificate. This file must be in PEM format.

    • Active Key Trustee Server TLS/SSL Server CA Certificate (PEM Format) (ssl.cacert.location)

      Default value: (none)

      The path to the file containing the CA certificate and any intermediate certificates (if any intermediate certificates exist, then they are required here) used to sign the Active Key Trustee Server certificate. If you have a CA-signed certificate, set this value to the path to the CA certificate or certificate chain file. This file must be in PEM format.

    • Active Key Trustee Server TLS/SSL Private Key Password (ssl.privatekey.password)

      Default value: (none)

      The password for the Active Key Trustee Server private key file. Leave this blank if the file is not password-protected.

    • Passive Key Trustee Server TLS/SSL Server Private Key File (PEM Format) (ssl.privatekey.location)

      Default value: /var/lib/keytrustee/.keytrustee/.ssl/ssl-cert-keytrustee-pk.pem

      The path to the Passive Key Trustee Server TLS certificate private key. Accept the default setting to use the auto-generated private key. If you have a CA-signed certificate, change this path to the CA-signed certificate private key file. This file must be in PEM format.

    • Passive Key Trustee Server TLS/SSL Server Certificate File (PEM Format) (ssl.cert.location)

      Default value: /var/lib/keytrustee/.keytrustee/.ssl/ssl-cert-keytrustee.pem

      The path to the Passive Key Trustee Server TLS certificate. Accept the default setting to use the auto-generated self-signed certificate. If you have a CA-signed certificate, change this to the path to the CA-signed certificate. This file must be in PEM format.

    • Passive Key Trustee Server TLS/SSL Server CA Certificate (PEM Format) (ssl.cacert.location)

      Default value: (none)

      The path to the file containing the CA certificate and any intermediate certificates (if any intermediate certificates exist, then they are required here) used to sign the Passive Key Trustee Server certificate. If you have a CA-signed certificate, set this value to the path to the CA certificate or certificate chain file. This file must be in PEM format.

    • Passive Key Trustee Server TLS/SSL Private Key Password (ssl.privatekey.password)

      Default value: (none)

      The password for the Passive Key Trustee Server private key file. Leave this blank if the file is not password-protected.

    After reviewing the settings and making any changes, click Continue.

  8. After all commands complete successfully, click Continue. If the Generate Key Trustee Server Keyring appears stuck, make sure that the Key Trustee Server host has enough entropy. See “Data at Rest Encryption Requirements” for more information.
  9. Click Finish to complete this step and return to the main page of the wizard.

8. Restart stale services and redeploy client configuration

Minimum Required Role: Cluster Administrator (also provided by Full Administrator) This feature is not available when using Cloudera Manager to manage Data Hub clusters.

This step restarts all services which were modified while enabling HDFS encryption. To complete this step:
  1. Click Restart stale services and redeploy client configuration.
  2. Click Restart Stale Services.
  3. Ensure that Re-deploy client configuration is checked, and click Restart Now.
  4. After all commands have completed, click Finish.

9. Validate Data Encryption

Minimum Required Role: Key Administrator or Cluster Administrator (also provided by Full Administrator) This feature is not available when using Cloudera Manager to manage Data Hub clusters.

This step launches a tutorial with instructions on creating an encryption zone and putting data into it to verify that HDFS encryption is enabled and working.

Enabling HDFS Encryption Using Navigator HSM KMS Backed by Thales HSM

1. Enable Kerberos

Minimum Required Role: Cluster Administrator (also provided by Full Administrator) This feature is not available when using Cloudera Manager to manage Data Hub clusters.

2. Enable TLS/SSL

Minimum Required Role: Cluster Administrator (also provided by Full Administrator) This feature is not available when using Cloudera Manager to manage Data Hub clusters.

3. Install the Thales HSM Client

Before installing the Navigator HSM KMS backed by Thales HSM, you must install the Thales HSM client on the host. Attempts to install the HSM KMS service before installing the Thales HSM client will fail.

For details about how to install the Thales HSM client, refer to the Thales HSM product documentation.

5. Add the HSM KMS backed by Thales Service

  1. Click Add Navigator HSM KMS Services backed by Thales HSM.
  2. In the Thales HSM KMS Proxy field, select the hosts to which you want to assign a new or existing role. Click OK, and then click Continue.
  3. To set up the ACL for the cluster, specify a comma-separated list of users and groups, and then click Generate ACLs. Click Continue.
  4. Click Continue.
  5. Review your selections and specify the:
    • Thales HSM Password

      Contact your HSM administrator for the Thales HSM password.

    • Keystore Password
    Then click Continue.
  6. Upon notification that you have successfully added the Thales KMS Service, click Continue and Finish.

6. Restart stale services and redeploy client configuration

Minimum Required Role: Cluster Administrator (also provided by Full Administrator) This feature is not available when using Cloudera Manager to manage Data Hub clusters.

This step restarts all services that were modified while enabling HDFS encryption. To complete this step:
  1. Click Restart stale services and redeploy client configuration.
  2. Click Restart Stale Services.
  3. Ensure that Re-deploy client configuration is checked, and click Restart Now.
  4. After all commands have completed, click Finish.

7. Validate Data Encryption

Minimum Required Role: Key Administrator or Cluster Administrator (also provided by Full Administrator) This feature is not available when using Cloudera Manager to manage Data Hub clusters.

This step launches a Validate Data Encryption tutorial with instructions describing how to create an encryption zone and place data into it to verify that HDFS encryption is enabled and working.

Enabling HDFS Encryption Using Navigator HSM KMS Backed by Luna HSM

1. Enable Kerberos

Minimum Required Role: Cluster Administrator (also provided by Full Administrator) This feature is not available when using Cloudera Manager to manage Data Hub clusters.

2. Enable TLS/SSL

Minimum Required Role: Cluster Administrator (also provided by Full Administrator) This feature is not available when using Cloudera Manager to manage Data Hub clusters.

3. Install Luna HSM Client

Before installing the Navigator HSM KMS backed by Luna HSM, you must install the Luna HSM client on the host. Attempts to install the Navigator HSM KMS backed by Luna HSM before installing the Luna HSM client will fail.

For details about how to install the Luna HSM client, refer to the Luna HSM product documentation.

4. Install Parcel for Cloudera Key Providers

Minimum Required Role: Cluster Administrator (also provided by Full Administrator) This feature is not available when using Cloudera Manager to manage Data Hub clusters.

This step completed automatically when you downloaded the parcel. If the step is incomplete for any reason (such as the wizard being interrupted or a failure installing the parcel), complete it manually:
  1. Click Install Key Trustee KMS binary using packages or parcels.
  2. Select the KEYTRUSTEE parcel to install Key Trustee KMS, or select None if you need to install Key Trustee KMS manually using packages. If you do not see a parcel available, click More Options and add the repository URL to the Remote Parcel Repository URLs list. After selecting a parcel, click Continue.
  3. After the KEYTRUSTEE parcel is successfully downloaded, distributed, unpacked, and activated, click Finish to complete this step and return to the main page of the wizard.

5. Add the Navigator HSM KMS backed by SafeNet Luna HSM

  1. Click Add Navigator HSM KMS backed by Safenet Luna HSM.
  2. In the Luna HSM-backed KMS Proxy field, select the hosts to which you want to assign a new or existing role. Click OK, and then click Continue.
  3. To set up the ACL for the cluster, specify a comma-separated list of users and groups, and then click Generate ACLs. Click Continue.
  4. Click Continue.
  5. Review your selections and specify the:
    • Luna HSM Password

      Contact your HSM administrator for the Luna HSM Partition password.

    • Keystore Password
    • Luna HSM Server Slot

      Identification number of the Luna HSM Server slot/device to use. If you do not know what value(s) to enter here, see the Luna product documentation for instructions on configuring your Luna HSM. Alternatively, run the /usr/safenet/lunaclient/bin/vtl verify command on the Luna HSM client host to view the slot value.

    Then click Continue.
  6. Upon notification that you have successfully added the Navigator Safenet Luna KMS Service, click Continue and Finish.

6. Restart stale services and redeploy client configuration

Minimum Required Role: Cluster Administrator (also provided by Full Administrator) This feature is not available when using Cloudera Manager to manage Data Hub clusters.

This step restarts all services that were modified while enabling HDFS encryption. To complete this step:
  1. Click Restart stale services and redeploy client configuration.
  2. Click Restart Stale Services.
  3. Ensure that Re-deploy client configuration is checked, and click Restart Now.
  4. After all commands have completed, click Finish.

7. Validate Data Encryption

Minimum Required Role: Key Administrator or Cluster Administrator (also provided by Full Administrator) This feature is not available when using Cloudera Manager to manage Data Hub clusters.

This step launches a Validate Data Encryption tutorial with instructions describing how to create an encryption zone and place data into it to verify that HDFS encryption is enabled and working.

Enabling HDFS Encryption Using a Java KeyStore

After selecting A file-based password-protected Java KeyStore as the root of trust, the following steps are displayed:

1. Enable Kerberos

Minimum Required Role: Cluster Administrator (also provided by Full Administrator) This feature is not available when using Cloudera Manager to manage Data Hub clusters.

2. Enable TLS/SSL

Minimum Required Role: Cluster Administrator (also provided by Full Administrator) This feature is not available when using Cloudera Manager to manage Data Hub clusters.

3. Add a Java KeyStore KMS Service

Minimum Required Role: Key Administrator (also provided by Full Administrator)

This step adds the Java KeyStore KMS service to the cluster. The Java KeyStore KMS service uses a password-protected Java KeyStore for cryptographic key management. To complete this step:
  1. Click Add a Java KeyStore KMS Service.
  2. Select a cluster host for the Java KeyStore KMS service. Click Continue.
  3. The Setup TLS for Java KeyStore KMS page provides high-level instructions for configuring TLS communication between the EDH cluster and the Java KeyStore KMS.

    Click Continue.

  4. The Review Changes page lists the Java KeyStore settings. Click the icon next to any setting for information about that setting. Enter the location and password for the Java KeyStore and click Continue.
  5. Click Continue to automatically configure the HDFS service to depend on the Java KeyStore KMS service.
  6. Click Finish to complete this step and return to the main page of the wizard.

4. Restart stale services and redeploy client configuration

Minimum Required Role: Cluster Administrator (also provided by Full Administrator) This feature is not available when using Cloudera Manager to manage Data Hub clusters.

This step restarts all services which were modified while enabling HDFS encryption. To complete this step:
  1. Click Restart stale services and redeploy client configuration.
  2. Click Restart Stale Services.
  3. Ensure that Re-deploy client configuration is checked, and click Restart Now.
  4. After all commands have completed, click Finish.

5. Validate Data Encryption

Minimum Required Role: Key Administrator or Cluster Administrator (also provided by Full Administrator) This feature is not available when using Cloudera Manager to manage Data Hub clusters.

This step launches a tutorial with instructions on creating an encryption zone and putting data into it to verify that HDFS encryption is enabled and working.

Hints and Tips

This section includes hints and tips that can help simplify the HSM KMS installation when using the HDFS Encryption Wizard.

Limit the Number of ZooKeeper DEBUG Messages

When setting the KMS log level to DEBUG, there can be a lot of ZooKeeper DEBUG messages that clutter the log. To prevent this, in the Logging Advanced Configuration Snippet (Safety Valve) field, enter:
log4j.category.org.apache.zookeeper=INFO

Limit Encryption Zone Timeouts

When creating encryption zones, there can be client timeouts due to the time it takes to fill the encrypted data encryption key (EDEK) cache. To avoid this, adjust the low watermark threshold settings as follows.

On the server side, in the field HSM KMS Proxy Advanced Configuration Snippet (Safety Valve) for kms-site.xml:
<property>
   <name>hadoop.security.kms.encrypted.key.cache.low.watermark</name>
   <value>.03</value>
</property>
On the client side, in the field HDFS Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml:
<property>
   <name>hadoop.security.kms.client.encrypted.key.cache.low-watermark</name>
   <value>.02</value>
</property>

Increase KMS Client Timeout Value

Due to potential latency during installation, it is recommended that you increase the KMS client timeout value.

Change from the default of 60 seconds to a value between 100 and 120 seconds in the field HDFS Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml:
<property>
   <name>hadoop.security.kms.client.timeout</name>
   <value>110</value>
</property>