Enabling HDFS Encryption Using the Wizard

To accommodate the security best practice of separation of duties, enabling HDFS encryption using the wizard requires different Cloudera Manager user roles for different steps.

Launch the Set up HDFS Data At Rest Encryption wizard in one of the following ways:
On the first page of the wizard, select the root of trust for encryption keys:
  • Cloudera Navigator Key Trustee Server
  • Navigator HSM KMS backed by Thales HSM
  • Navigator HSM KMS backed by Luna HSM
  • A file-based password-protected Java KeyStore
Cloudera strongly recommends using Cloudera Navigator Key Trustee Server as the root of trust for production environments. The file-based Java KeyStore root of trust is insufficient to provide the security, scalability, and manageability required by most production systems. More specifically, the Java KeyStore KMS does not provide:
  • Scalability, so you are limited to only one KMS, which can result in bottlenecks
  • High Availability (HA)
  • Recoverability, so if you lose the node where the Java KeyStore is stored, then you can lose access to all the encrypted data

Ultimately, the Java KeyStore does not satisfy the stringent security requirements of most organizations for handling master encryption keys.

Choosing a root of trust displays a list of steps required to enable HDFS encryption using that root of trust. Each step can be completed independently. The Status column indicates whether the step has been completed, and the Notes column provides additional context for the step. If your Cloudera Manager user account does not have sufficient privileges to complete a step, the Notes column indicates the required privileges.

Available steps contain links to wizards or documentation required to complete the step. If a step is unavailable due to insufficient privileges or a prerequisite step being incomplete, no links are present and the Notes column indicates the reason the step is unavailable.

Enabling HDFS Encryption Using Navigator Key Trustee Server

Enabling HDFS encryption using Key Trustee Server as the key store involves multiple components. For an overview of the components involved in encrypting data at rest, see Cloudera Navigator Data Encryption Overview. For guidelines on deploying the Navigator Key Trustee Server in production environments, Resource Planning for Data at Rest Encryption.

Before continuing, make sure the Cloudera Manager server host has access to the internal repository hosting the Key Trustee Server software. See Setting Up an Internal Repository for more information.

1. Enable Kerberos

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

For more information about enabling Kerberos, see Enabling Kerberos Authentication Using the Wizard.

2. Enable TLS/SSL

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

For more information about enabling TLS, see Configuring TLS/SSL Encryption for CDH Services.

3. Add a dedicated cluster for the Key Trustee Server

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

If you haven't already done so, you must create an internal repository to install Cloudera Navigator before you can set up and use Navigator Key Trustee Server. For instructions on creating internal repositories (including Cloudera Manager, CDH, and Cloudera Navigator encryption components), see Using an Internal Parcel Repository and Using an Internal Package Repository.

This step creates a new cluster in Cloudera Manager for the Key Trustee Server hosts to isolate them from other enterprise data hub (EDH) services for increased security and durability. For more information, see Data at Rest Encryption Reference Architecture.

To complete this step:

  1. Click Add a dedicated cluster for the Key Trustee Server.
  2. Leave Enable High Availability checked to add two hosts to the cluster. For production environments, you must enable high availability for Key Trustee Server. Failure to enable high availability can result in complete data loss in the case of catastrophic failure of a standalone Key Trustee Server. Click Continue.
  3. Search for new hosts to add to the cluster, or select the Currently Managed Hosts tab to add existing hosts to the cluster. After selecting the hosts, click Continue.
  4. Select the KEYTRUSTEE_SERVER parcel to install Key Trustee Server using parcels, or select None if you want to use packages. If you do not see a parcel available, click More Options and add the repository URL to the Remote Parcel Repository URLs list. After selecting a parcel or None, click Continue.

    If you selected None, click Continue again, and skip to 4. Install Key Trustee Server binary using packages or parcels.

  5. After the KEYTRUSTEE_SERVER parcel is successfully downloaded, distributed, unpacked, and activated, click Continue.
  6. Click Continue to complete this step and return to the main page of the wizard.

4. Install Key Trustee Server binary using packages or parcels

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

If you haven't already done so, you must create an internal repository to install Cloudera Navigator before you can set up and use Navigator Key Trustee Server. For instructions on creating internal repositories (including Cloudera Manager, CDH, and Cloudera Navigator encryption components), see Using an Internal Parcel Repository and Using an Internal Package Repository.

This step is completed automatically during 3. Add a dedicated cluster for the Key Trustee Server if you are using parcels. If the step is incomplete for any reason (such as the wizard being interrupted or a failure installing the parcel), complete it manually:
  1. Click Install Key Trustee Server binary using packages or parcels.
  2. Select the KEYTRUSTEE_SERVER parcel to install Key Trustee Server, or select None if you need to install Key Trustee Server manually using packages. If you do not see a parcel available, click More Options and add the repository URL to the Remote Parcel Repository URLs list. After selecting a parcel, click Continue.
  3. After the KEYTRUSTEE_SERVER parcel is successfully downloaded, distributed, unpacked, and activated, click Finish to complete this step and return to the main page of the wizard.

5. Install Parcel for Key Trustee KMS

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

If you haven't already done so, you must create an internal repository to install Cloudera Navigator before you can set up and use Navigator Key Trustee Server. For instructions on creating internal repositories (including Cloudera Manager, CDH, and Cloudera Navigator encryption components), see Using an Internal Parcel Repository and Using an Internal Package Repository.

This step installs the Key Trustee KMS parcel. If you are using packages, skip this step and see Installing Key Trustee KMS Using Packages for instructions. After installing Key Trustee KMS using packages, continue to 6. Add a Key Trustee Server Service.

To complete this step for parcel-based installations:
  1. Click Install Parcel for Key Trustee KMS.
  2. Select the KEYTRUSTEE parcel to install Key Trustee KMS. If you do not see a parcel available, click More Options and add the repository URL to the Remote Parcel Repository URLs list. After selecting a parcel, click Continue.
  3. After the KEYTRUSTEE parcel is successfully downloaded, distributed, unpacked, and activated, click Finish to complete this step and return to the main page of the wizard.

6. Add a Key Trustee Server Service

Minimum Required Role: Key Administrator (also provided by Full Administrator)

This step adds the Key Trustee Server service to Cloudera Manager. To complete this step:

  1. Click Add a Key Trustee Server Service.
  2. Click Continue.
  3. On the Customize Role Assignments for Key Trustee Server page, select the hosts for the Active Key Trustee Server and Passive Key Trustee Server roles. Make sure that the selected hosts are not used for other services (see Resource Planning for Data at Rest Encryption for more information), and click Continue.
  4. The Entropy Considerations page provides commands to install the rng-tools package to increase available entropy for cryptographic operations. For more information, see Entropy requirements in Data at Rest Encryption Requirements. After completing these commands, click Continue.
  5. The Synchronize Active and Passive Key Trustee Server Private Keys page provides instructions for generating and copying the Active Key Trustee Server private key to the Passive Key Trustee Server. Cloudera recommends following security best practices and transferring the private key using offline media, such as a removable USB drive. For convenience (for example, in a development or testing environment where maximum security is not required), you can copy the private key over the network using the provided rsync command.

    After you have synchronized the private keys, run the ktadmin init command on the Passive Key Trustee Server as described in the wizard. After the initialization is complete, check the box to indicate you have synchronized the keys and click Continue in the wizard.

  6. The Setup TLS for Key Trustee Server page provides instructions on replacing the auto-generated self-signed certificate with a production certificate from a trusted Certificate Authority (CA). For more information, see Managing Key Trustee Server Certificates. Click Continue to view and modify the default certificate settings.
  7. On the Review Changes page, you can view and modify the following settings:
    • Database Storage Directory (db_root)

      Default value: /var/lib/keytrustee/db

      The directory on the local filesystem where the Key Trustee Server database is stored. Modify this value to store the database in a different directory.

    • Active Key Trustee Server TLS/SSL Server Private Key File (PEM Format) (ssl.privatekey.location)

      Default value: /var/lib/keytrustee/.keytrustee/.ssl/ssl-cert-keytrustee-pk.pem

      The path to the Active Key Trustee Server TLS certificate private key. Accept the default setting to use the auto-generated private key. If you have a CA-signed certificate, change this path to the CA-signed certificate private key file. This file must be in PEM format.

    • Active Key Trustee Server TLS/SSL Server Certificate File (PEM Format) (ssl.cert.location)

      Default value: /var/lib/keytrustee/.keytrustee/.ssl/ssl-cert-keytrustee.pem

      The path to the Active Key Trustee Server TLS certificate. Accept the default setting to use the auto-generated self-signed certificate. If you have a CA-signed certificate, change this to the path to the CA-signed certificate. This file must be in PEM format.

    • Active Key Trustee Server TLS/SSL Server CA Certificate (PEM Format) (ssl.cacert.location)

      Default value: (none)

      The path to the file containing the CA certificate and any intermediate certificates (if any intermediate certificates exist, then they are required here) used to sign the Active Key Trustee Server certificate. If you have a CA-signed certificate, set this value to the path to the CA certificate or certificate chain file. This file must be in PEM format.

    • Active Key Trustee Server TLS/SSL Private Key Password (ssl.privatekey.password)

      Default value: (none)

      The password for the Active Key Trustee Server private key file. Leave this blank if the file is not password-protected.

    • Passive Key Trustee Server TLS/SSL Server Private Key File (PEM Format) (ssl.privatekey.location)

      Default value: /var/lib/keytrustee/.keytrustee/.ssl/ssl-cert-keytrustee-pk.pem

      The path to the Passive Key Trustee Server TLS certificate private key. Accept the default setting to use the auto-generated private key. If you have a CA-signed certificate, change this path to the CA-signed certificate private key file. This file must be in PEM format.

    • Passive Key Trustee Server TLS/SSL Server Certificate File (PEM Format) (ssl.cert.location)

      Default value: /var/lib/keytrustee/.keytrustee/.ssl/ssl-cert-keytrustee.pem

      The path to the Passive Key Trustee Server TLS certificate. Accept the default setting to use the auto-generated self-signed certificate. If you have a CA-signed certificate, change this to the path to the CA-signed certificate. This file must be in PEM format.

    • Passive Key Trustee Server TLS/SSL Server CA Certificate (PEM Format) (ssl.cacert.location)

      Default value: (none)

      The path to the file containing the CA certificate and any intermediate certificates (if any intermediate certificates exist, then they are required here) used to sign the Passive Key Trustee Server certificate. If you have a CA-signed certificate, set this value to the path to the CA certificate or certificate chain file. This file must be in PEM format.

    • Passive Key Trustee Server TLS/SSL Private Key Password (ssl.privatekey.password)

      Default value: (none)

      The password for the Passive Key Trustee Server private key file. Leave this blank if the file is not password-protected.

    After reviewing the settings and making any changes, click Continue.

  8. After all commands complete successfully, click Continue. If the Generate Key Trustee Server Keyring appears stuck, make sure that the Key Trustee Server host has enough entropy. See entropy requirements in Data at Rest Encryption Requirements for more information.
  9. Click Finish to complete this step and return to the main page of the wizard.

For parcel-based Key Trustee Server releases 5.8 and higher, Cloudera Manager automatically backs up Key Trustee Server (using the ktbackup.sh script) after adding the Key Trustee Server service. It also schedules automatic backups using cron. For package-based installations, you must manually back up Key Trustee Server and configure a cron job.

Cloudera Manager configures cron to run the backup script hourly. The latest 10 backups are retained in /var/lib/keytrustee in cleartext. For information about using the backup script and configuring the cron job (including how to encrypt backups), see Backing Up Key Trustee Server and Key Trustee KMS Using the ktbackup.sh Script.

7. Add a Key Trustee KMS Service

Minimum Required Role: Key Administrator (also provided by Full Administrator)

This step adds a Key Trustee KMS service to the cluster. The Key Trustee KMS service is required to enable HDFS encryption to use Key Trustee Server for cryptographic key management. Key Trustee KMS high availability uses ZooKeeper to automatically configure load balancing. If you do not have a ZooKeeper service in your cluster, add one using the instructions in Adding a Service.

To complete this step:

  1. Click Add a Key Trustee KMS Service.
  2. Select an existing Key Trustee Server pair or specify an external Key Trustee Server pair. If you have an existing Key Trustee Server pair outside of Cloudera Manager control, select the External Key Trustee Server option and specify the fully-qualified domain names (FQDNs) of the Key Trustee Server pair. Click Continue.
  3. Select cluster hosts for the Key Trustee KMS service. For production environments, select at least two hosts for high availability. If you proceed with only one host, you can enable high availability later. See Enabling Key Trustee KMS High Availability for more information.

    Make sure that the selected hosts are not used for other services (see Resource Planning for Data at Rest Encryption for more information), and click Continue.

  4. The Entropy Considerations page provides commands to install the rng-tools package to increase available entropy for cryptographic operations. For more information, see entropy requirements in Data at Rest Encryption Requirements. After completing these commands, click Continue.
  5. The Setup Organization and Auth Secret page generates the necessary commands to create an organization in Key Trustee Server. An organization is required to be able to register the Key Trustee KMS with Key Trustee Server. See Managing Key Trustee Server Organizations for more information.

    Enter an organization name and click Generate Instruction. Run the displayed commands to generate an organization and obtain the auth_secret value for the organization. Enter the secret in the auth_secret field and click Continue.

  6. The Setup Access Control List (ACL) page allows you to generate ACLs for the Key Trustee KMS or to provide your own ACLs. To generate the recommended ACLs, enter the username and group responsible for managing cryptographic keys and click Generate ACLs. To specify your own ACLs, select the Use Your Own kms-acls.xml File option and enter the ACLs. For more information on the KMS Access Control List, see Configuring KMS Access Control Lists (ACLs).

    After generating or specifying the ACL, click Continue.

  7. The Setup TLS for Key Trustee KMS page provides high-level instructions for configuring TLS communication between the Key Trustee KMS and the Key Trustee Server, as well as between the EDH cluster and the Key Trustee KMS. See Configuring TLS/SSL for the KMS for more information.

    Click Continue.

  8. The Review Changes page lists all of the settings configured in this step. Click the question mark icon next to any setting for information about that setting. Review the settings and click Continue.
  9. After the First Run commands have successfully completed, click Continue.
  10. The Synchronize Private Keys and HDFS Dependency page provides instructions for copying the private key from one Key Management Server Proxy role to all other roles.

    Cloudera recommends following security best practices and transferring the private key using offline media, such as a removable USB drive. For convenience (for example, in a development or testing environment where maximum security is not required), you can copy the private key over the network using the provided rsync command.

    After you have synchronized the private keys, check the box to indicate you have done so and click Continue.

  11. After the Key Trustee KMS service starts, click Finish to complete this step and return to the main page of the wizard.

For parcel-based Key Trustee KMS releases 5.8 and higher, Cloudera Manager automatically backs up Key Trustee KMS (using the ktbackup.sh script) after adding the Key Trustee KMS service. It does not schedule automatic backups using cron. For package-based installations, you must manually back up Key Trustee Server and configure a cron job.

The backup is stored in /var/lib/kms-keytrustee in cleartext. For more information about using the backup script and configuring the cron job (including how to encrypt backups), see Backing Up Key Trustee Server and Key Trustee KMS Using the ktbackup.sh Script.

8. Restart stale services and redeploy client configuration

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

This step restarts all services which were modified while enabling HDFS encryption. To complete this step:
  1. Click Restart stale services and redeploy client configuration.
  2. Click Restart Stale Services.
  3. Ensure that Re-deploy client configuration is checked, and click Restart Now.
  4. After all commands have completed, click Finish.

9. Validate Data Encryption

Minimum Required Role: Key Administrator or Cluster Administrator (also provided by Full Administrator)

This step launches a tutorial with instructions on creating an encryption zone and putting data into it to verify that HDFS encryption is enabled and working.

Enabling HDFS Encryption Using Navigator HSM KMS Backed by Thales HSM

1. Enable Kerberos

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

For more information about enabling Kerberos, see Enabling Kerberos Authentication Using the Wizard.

2. Enable TLS/SSL

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

For more information about enabling TLS, see Configuring TLS Encryption for Cloudera Manager.

3. Install the Thales HSM Client

Before installing the Navigator HSM KMS backed by Thales HSM, you must install the Thales HSM client on the host. Attempts to install the HSM KMS service before installing the Thales HSM client will fail.

For details about how to install the Thales HSM client, refer to the Thales HSM product documentation.

4. Install Key Trustee KMS binary using parcels

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

This step completes automatically when you download the parcel. If the step is incomplete for any reason (such as the wizard being interrupted or a failure installing the parcel), complete it manually:
  1. Click Install Key Trustee KMS binary using parcels.
  2. Select the KEYTRUSTEE parcel to install Key Trustee KMS, or select None if you need to install Key Trustee KMS manually using packages. If you do not see a parcel available, click More Options and add the repository URL to the Remote Parcel Repository URLs list. After selecting a parcel, click Continue.
  3. After the KEYTRUSTEE parcel is successfully downloaded, distributed, unpacked, and activated, click Finish to complete this step and return to the main page of the wizard.

5. Add the HSM KMS backed by Thales Service

  1. Click Add Navigator HSM KMS Services backed by Thales HSM.
  2. In the Thales HSM KMS Proxy field, select the hosts to which you want to assign a new or existing role. Click OK, and then click Continue.
  3. To set up the ACL for the cluster, specify a comma-separated list of users and groups, and then click Generate ACLs. Click Continue.
  4. Click Continue.
  5. Review your selections and specify the:
    • Thales HSM Password

      Contact your HSM administrator for the Thales HSM password.

    • Keystore Password
    Then click Continue.
  6. Upon notification that you have successfully added the Thales KMS Service, click Continue and Finish.

6. Restart stale services and redeploy client configuration

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

This step restarts all services that were modified while enabling HDFS encryption. To complete this step:
  1. Click Restart stale services and redeploy client configuration.
  2. Click Restart Stale Services.
  3. Ensure that Re-deploy client configuration is checked, and click Restart Now.
  4. After all commands have completed, click Finish.

7. Validate Data Encryption

Minimum Required Role: Key Administrator or Cluster Administrator (also provided by Full Administrator)

This step launches a Validate Data Encryption tutorial with instructions describing how to create an encryption zone and place data into it to verify that HDFS encryption is enabled and working.

Enabling HDFS Encryption Using Navigator HSM KMS Backed by Luna HSM

1. Enable Kerberos

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

For more information about enabling Kerberos, see Enabling Kerberos Authentication Using the Wizard.

2. Enable TLS/SSL

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

For more information about enabling TLS, see Configuring TLS Encryption for Cloudera Manager.

3. Install Luna HSM Client

Before installing the Navigator HSM KMS backed by Luna HSM, you must install the Luna HSM client on the host. Attempts to install the Navigator HSM KMS backed by Luna HSM before installing the Luna HSM client will fail.

For details about how to install the Luna HSM client, refer to the Luna HSM product documentation.

4. Install Parcel for Cloudera Key Providers

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

This step completed automatically when you downloaded the parcel. If the step is incomplete for any reason (such as the wizard being interrupted or a failure installing the parcel), complete it manually:
  1. Click Install Key Trustee KMS binary using packages or parcels.
  2. Select the KEYTRUSTEE parcel to install Key Trustee KMS, or select None if you need to install Key Trustee KMS manually using packages. If you do not see a parcel available, click More Options and add the repository URL to the Remote Parcel Repository URLs list. After selecting a parcel, click Continue.
  3. After the KEYTRUSTEE parcel is successfully downloaded, distributed, unpacked, and activated, click Finish to complete this step and return to the main page of the wizard.

5. Add the Navigator HSM KMS backed by SafeNet Luna HSM

  1. Click Add Navigator HSM KMS backed by Safenet Luna HSM.
  2. In the Luna HSM-backed KMS Proxy field, select the hosts to which you want to assign a new or existing role. Click OK, and then click Continue.
  3. To set up the ACL for the cluster, specify a comma-separated list of users and groups, and then click Generate ACLs. Click Continue.
  4. Click Continue.
  5. Review your selections and specify the:
    • Luna HSM Password

      Contact your HSM administrator for the Luna HSM Partition password.

    • Keystore Password
    • Luna HSM Server Slot

      Identification number of the Luna HSM Server slot/device to use. If you do not know what value(s) to enter here, see the Luna product documentation for instructions on configuring your Luna HSM. Alternatively, run the /usr/safenet/lunaclient/bin/vtl verify command on the Luna HSM client host to view the slot value.

    Then click Continue.
  6. Upon notification that you have successfully added the Navigator Safenet Luna KMS Service, click Continue and Finish.

6. Restart stale services and redeploy client configuration

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

This step restarts all services that were modified while enabling HDFS encryption. To complete this step:
  1. Click Restart stale services and redeploy client configuration.
  2. Click Restart Stale Services.
  3. Ensure that Re-deploy client configuration is checked, and click Restart Now.
  4. After all commands have completed, click Finish.

7. Validate Data Encryption

Minimum Required Role: Key Administrator or Cluster Administrator (also provided by Full Administrator)

This step launches a Validate Data Encryption tutorial with instructions describing how to create an encryption zone and place data into it to verify that HDFS encryption is enabled and working.

Enabling HDFS Encryption Using a Java KeyStore

After selecting A file-based password-protected Java KeyStore as the root of trust, the following steps are displayed:

1. Enable Kerberos

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

For more information on enabling Kerberos, see Enabling Kerberos Authentication Using the Wizard.

2. Enable TLS/SSL

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

For more information on enabling TLS, see Configuring TLS Encryption for Cloudera Manager.

3. Add a Java KeyStore KMS Service

Minimum Required Role: Key Administrator (also provided by Full Administrator)

This step adds the Java KeyStore KMS service to the cluster. The Java KeyStore KMS service uses a password-protected Java KeyStore for cryptographic key management. To complete this step:
  1. Click Add a Java KeyStore KMS Service.
  2. Select a cluster host for the Java KeyStore KMS service. Click Continue.
  3. The Setup TLS for Java KeyStore KMS page provides high-level instructions for configuring TLS communication between the EDH cluster and the Java KeyStore KMS. See Configuring TLS/SSL for the KMS for more information.

    Click Continue.

  4. The Review Changes page lists the Java KeyStore settings. Click the question mark icon next to any setting for information about that setting. Enter the location and password for the Java KeyStore and click Continue.
  5. Click Continue to automatically configure the HDFS service to depend on the Java KeyStore KMS service.
  6. Click Finish to complete this step and return to the main page of the wizard.

4. Restart stale services and redeploy client configuration

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

This step restarts all services which were modified while enabling HDFS encryption. To complete this step:
  1. Click Restart stale services and redeploy client configuration.
  2. Click Restart Stale Services.
  3. Ensure that Re-deploy client configuration is checked, and click Restart Now.
  4. After all commands have completed, click Finish.

5. Validate Data Encryption

Minimum Required Role: Key Administrator or Cluster Administrator (also provided by Full Administrator)

This step launches a tutorial with instructions on creating an encryption zone and putting data into it to verify that HDFS encryption is enabled and working.

Hints and Tips

This section includes hints and tips that can help simplify the HSM KMS installation when using the HDFS Encryption Wizard.

Limit the Number of ZooKeeper DEBUG Messages

When setting the KMS log level to DEBUG, there can be a lot of ZooKeeper DEBUG messages that clutter the log. To prevent this, in the Logging Advanced Configuration Snippet (Safety Valve) field, enter:
log4j.category.org.apache.zookeeper=INFO

Limit Encryption Zone Timeouts

When creating encryption zones, there can be client timeouts due to the time it takes to fill the encrypted data encryption key (EDEK) cache. To avoid this, adjust the low watermark threshold settings as follows.

On the server side, in the field HSM KMS Proxy Advanced Configuration Snippet (Safety Valve) for kms-site.xml:
<property>
   <name>hadoop.security.kms.encrypted.key.cache.low.watermark</name>
   <value>.03</value>
</property>
On the client side, in the field HDFS Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml:
<property>
   <name>hadoop.security.kms.client.encrypted.key.cache.low-watermark</name>
   <value>.02</value>
</property>

Increase KMS Client Timeout Value

Due to potential latency during installation, it is recommended that you increase the KMS client timeout value.

Change from the default of 60 seconds to a value between 100 and 120 seconds in the field HDFS Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml:
<property>
   <name>hadoop.security.kms.client.timeout</name>
   <value>110</value>
</property>