Enabling HDFS Encryption Using the Wizard

To accommodate the security best practice of separation of duties, enabling HDFS encryption using the wizard requires different Cloudera Manager user roles for different steps.

Launch the Set up HDFS Data At Rest Encryption wizard in one of the following ways:
On the first page of the wizard, select the root of trust for encryption keys:
  • Cloudera Navigator Key Trustee Server
  • A file-based password-protected Java KeyStore
Cloudera strongly recommends using Cloudera Navigator Key Trustee Server as the root of trust for production environments. The file-based Java KeyStore root of trust is insufficient to provide the security, scalability, and manageability required by most production systems. More specifically, the Java KeyStore KMS does not provide:
  • Scalability, so you are limited to only one KMS, which can result in bottlenecks
  • High Availability (HA)
  • Recoverability, so if you lose the node where the Java KeyStore is stored, then you can lose access to all the encrypted data

Ultimately, the Java KeyStore does not satisfy the stringent security requirements of most organizations for handling master encryption keys.

Choosing a root of trust displays a list of steps required to enable HDFS encryption using that root of trust. Each step can be completed independently. The Status column indicates whether the step has been completed, and the Notes column provides additional context for the step. If your Cloudera Manager user account does not have sufficient privileges to complete a step, the Notes column indicates the required privileges.

Available steps contain links to wizards or documentation required to complete the step. If a step is unavailable due to insufficient privileges or a prerequisite step being incomplete, no links are present and the Notes column indicates the reason the step is unavailable.

Continue to the section for your selected root of trust for further instructions:

Enabling HDFS Encryption Using Cloudera Navigator Key Trustee Server

Enabling HDFS encryption using Key Trustee Server as the key store involves multiple components. For an overview of the components involved in encrypting data at rest, see Cloudera Navigator Data Encryption Overview.

Before continuing, make sure the Cloudera Manager server host has access to the internal repository hosting the Key Trustee Server software. See Setting Up an Internal Repository for more information.

After selecting Cloudera Navigator Key Trustee Server as the root of trust, the following steps are displayed:

1. Enable Kerberos

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

For more information about enabling Kerberos, see Enabling Kerberos Authentication Using the Wizard.

2. Enable TLS/SSL

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

For more information about enabling TLS, see Configuring TLS/SSL Encryption for CDH Services.

3. Add a dedicated cluster for the Key Trustee Server

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

This step creates a new cluster in Cloudera Manager for the Key Trustee Server hosts to isolate them from other enterprise data hub (EDH) services for increased security and durability. For more information, see Resource Planning for Data at Rest Encryption and Data at Rest Encryption Reference Architecture.

To complete this step:

  1. Click Add a dedicated cluster for the Key Trustee Server.
  2. Leave Enable High Availability checked to add two hosts to the cluster. Cloudera strongly recommends using high availability for Key Trustee Server. Failure to enable high availability can result in complete data loss in the case of catastrophic failure of a standalone Key Trustee Server. Click Continue.
  3. Search for new hosts to add to the cluster, or select the Currently Managed Hosts tab to add existing hosts to the cluster. After selecting the hosts, click Continue.
  4. Select the KEYTRUSTEE_SERVER parcel to install Key Trustee Server using parcels, or select None if you want to use packages. If you do not see a parcel available, click More Options and add the repository URL to the Remote Parcel Repository URLs list. After selecting a parcel or None, click Continue.

    If you selected None, click Continue again, and skip to 4. Install Key Trustee Server binary using packages or parcels.

  5. After the KEYTRUSTEE_SERVER parcel is successfully downloaded, distributed, unpacked, and activated, click Continue.
  6. Click Continue to complete this step and return to the main page of the wizard.

4. Install Key Trustee Server binary using packages or parcels

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

This step is completed automatically during 3. Add a dedicated cluster for the Key Trustee Server if you are using parcels. If the step is incomplete for any reason (such as the wizard being interrupted or a failure installing the parcel), complete it manually:
  1. Click Install Key Trustee Server binary using packages or parcels.
  2. Select the KEYTRUSTEE_SERVER parcel to install Key Trustee Server, or select None if you need to install Key Trustee Server manually using packages. If you do not see a parcel available, click More Options and add the repository URL to the Remote Parcel Repository URLs list. After selecting a parcel, click Continue.
  3. After the KEYTRUSTEE_SERVER parcel is successfully downloaded, distributed, unpacked, and activated, click Finish to complete this step and return to the main page of the wizard.

5. Install Parcel for Key Trustee KMS

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

This step installs the Key Trustee KMS parcel. If you are using packages, skip this step and see Installing Key Trustee KMS Using Packages for instructions. After installing Key Trustee KMS using packages, continue to 6. Add a Key Trustee Server Service.

To complete this step for parcel-based installations:
  1. Click Install Parcel for Key Trustee KMS.
  2. Select the KEYTRUSTEE parcel to install Key Trustee KMS. If you do not see a parcel available, click More Options and add the repository URL to the Remote Parcel Repository URLs list. After selecting a parcel, click Continue.
  3. After the KEYTRUSTEE parcel is successfully downloaded, distributed, unpacked, and activated, click Finish to complete this step and return to the main page of the wizard.

6. Add a Key Trustee Server Service

Minimum Required Role: Key Administrator (also provided by Full Administrator)

This step adds the Key Trustee Server service to Cloudera Manager. To complete this step:
  1. Click Add a Key Trustee Server Service.
  2. Click Continue.
  3. On the Customize Role Assignments for Key Trustee Server page, select the hosts for the Active Key Trustee Server and Passive Key Trustee Server roles. Make sure that the selected hosts are not used for other services (see Resource Planning for Data at Rest Encryption for more information), and click Continue.
  4. The Entropy Considerations page provides commands to install the rng-tools package to increase available entropy for cryptographic operations. For more information, see Entropy Requirements. After completing these commands, click Continue.
  5. The Synchronize Active and Passive Key Trustee Server Private Keys page provides instructions for generating and copying the Active Key Trustee Server private key to the Passive Key Trustee Server. Cloudera recommends following security best practices and transferring the private key using offline media, such as a removable USB drive. For convenience (for example, in a development or testing environment where maximum security is not required), you can copy the private key over the network using the provided rsync command.

    After you have synchronized the private keys, run the ktadmin init command on the Passive Key Trustee Server as described in the wizard. After the initialization is complete, check the box to indicate you have synchronized the keys and click Continue in the wizard.

  6. The Setup TLS for Key Trustee Server page provides instructions on replacing the auto-generated self-signed certificate with a production certificate from a trusted Certificate Authority (CA). For more information, see Managing Key Trustee Server Certificates. Click Continue to view and modify the default certificate settings.
  7. On the Review Changes page, you can view and modify the following settings:
    • Database Storage Directory (db_root)

      Default value: /var/lib/keytrustee/db

      The directory on the local filesystem where the Key Trustee Server database is stored. Modify this value to store the database in a different directory.

    • Active Key Trustee Server TLS/SSL Server Private Key File (PEM Format) (ssl.privatekey.location)

      Default value: /var/lib/keytrustee/.keytrustee/.ssl/ssl-cert-keytrustee-pk.pem

      The path to the Active Key Trustee Server TLS certificate private key. Accept the default setting to use the auto-generated private key. If you have a CA-signed certificate, change this path to the CA-signed certificate private key file. This file must be in PEM format.

    • Active Key Trustee Server TLS/SSL Server Certificate File (PEM Format) (ssl.cert.location)

      Default value: /var/lib/keytrustee/.keytrustee/.ssl/ssl-cert-keytrustee.pem

      The path to the Active Key Trustee Server TLS certificate. Accept the default setting to use the auto-generated self-signed certificate. If you have a CA-signed certificate, change this to the path to the CA-signed certificate. This file must be in PEM format.

    • Active Key Trustee Server TLS/SSL Server CA Certificate (PEM Format) (ssl.cacert.location)

      Default value: (none)

      The path to the file containing the CA certificate and any intermediate certificates (if any intermediate certificates exist, then they are required here) used to sign the Active Key Trustee Server certificate. If you have a CA-signed certificate, set this value to the path to the CA certificate or certificate chain file. This file must be in PEM format.

    • Active Key Trustee Server TLS/SSL Private Key Password (ssl.privatekey.password)

      Default value: (none)

      The password for the Active Key Trustee Server private key file. Leave this blank if the file is not password-protected.

    • Passive Key Trustee Server TLS/SSL Server Private Key File (PEM Format) (ssl.privatekey.location)

      Default value: /var/lib/keytrustee/.keytrustee/.ssl/ssl-cert-keytrustee-pk.pem

      The path to the Passive Key Trustee Server TLS certificate private key. Accept the default setting to use the auto-generated private key. If you have a CA-signed certificate, change this path to the CA-signed certificate private key file. This file must be in PEM format.

    • Passive Key Trustee Server TLS/SSL Server Certificate File (PEM Format) (ssl.cert.location)

      Default value: /var/lib/keytrustee/.keytrustee/.ssl/ssl-cert-keytrustee.pem

      The path to the Passive Key Trustee Server TLS certificate. Accept the default setting to use the auto-generated self-signed certificate. If you have a CA-signed certificate, change this to the path to the CA-signed certificate. This file must be in PEM format.

    • Passive Key Trustee Server TLS/SSL Server CA Certificate (PEM Format) (ssl.cacert.location)

      Default value: (none)

      The path to the file containing the CA certificate and any intermediate certificates (if any intermediate certificates exist, then they are required here) used to sign the Passive Key Trustee Server certificate. If you have a CA-signed certificate, set this value to the path to the CA certificate or certificate chain file. This file must be in PEM format.

    • Passive Key Trustee Server TLS/SSL Private Key Password (ssl.privatekey.password)

      Default value: (none)

      The password for the Passive Key Trustee Server private key file. Leave this blank if the file is not password-protected.

    After reviewing the settings and making any changes, click Continue.

  8. After all commands complete successfully, click Continue. If the Generate Key Trustee Server Keyring appears stuck, make sure that the Key Trustee Server host has enough entropy. See Entropy Requirements for more information.
  9. Click Finish to complete this step and return to the main page of the wizard.

7. Add a Key Trustee KMS Service

Minimum Required Role: Key Administrator (also provided by Full Administrator)

This step adds a Key Trustee KMS service to the cluster. The Key Trustee KMS service is required to enable HDFS encryption to use Key Trustee Server for cryptographic key management. To complete this step:
  1. Click Add a Key Trustee KMS Service.
  2. Select an existing Key Trustee Server pair or specify an external Key Trustee Server pair. If you have an existing Key Trustee Server pair outside of Cloudera Manager control, select the External Key Trustee Server option and specify the fully-qualified domain names (FQDNs) of the Key Trustee Server pair. Click Continue.
  3. Select cluster hosts for the Key Trustee KMS service. Cloudera recommends selecting at least two hosts for high availability. If you proceed with only one host, you can enable high availability later. See Key Trustee KMS High Availability for more information.

    Make sure that the selected hosts are not used for other services (see Resource Planning for Data at Rest Encryption for more information), and click Continue.

  4. The Entropy Considerations page provides commands to install the rng-tools package to increase available entropy for cryptographic operations. For more information, see Entropy Requirements. After completing these commands, click Continue.
  5. The Setup Organization and Auth Secret page generates the necessary commands to create an organization in Key Trustee Server. An organization is required to be able to register the Key Trustee KMS with Key Trustee Server. See Managing Key Trustee Server Organizations for more information.

    Enter an organization name and click Generate Instruction. Run the displayed commands to generate an organization and obtain the auth_secret value for the organization. Enter the secret in the auth_secret field and click Continue.

  6. The Setup Access Control List (ACL) page allows you to generate ACLs for the Key Trustee KMS or to provide your own ACLs. To generate the recommended ACLs, enter the username and group responsible for managing cryptographic keys and click Generate ACLs. To specify your own ACLs, select the Use Your Own kms-acls.xml File option and enter the ACLs. For more information on the KMS Access Control List, see Configuring KMS Access Control Lists.

    After generating or specifying the ACL, click Continue.

  7. The Setup TLS for Key Trustee KMS page provides high-level instructions for configuring TLS communication between the Key Trustee KMS and the Key Trustee Server, as well as between the EDH cluster and the Key Trustee KMS. See Configuring TLS/SSL for the KMS for more information.

    Click Continue.

  8. The Review Changes page lists all of the settings configured in this step. Click the icon next to any setting for information about that setting. Review the settings and click Continue.
  9. After the First Run commands have successfully completed, click Continue.
  10. The Synchronize Private Keys and HDFS Dependency page provides instructions for copying the private key from one Key Management Server Proxy role to all other roles.

    Cloudera recommends following security best practices and transferring the private key using offline media, such as a removable USB drive. For convenience (for example, in a development or testing environment where maximum security is not required), you can copy the private key over the network using the provided rsync command.

    After you have synchronized the private keys, check the box to indicate you have done so and click Continue.

  11. After the Key Trustee KMS service starts, click Finish to complete this step and return to the main page of the wizard.

8. Restart stale services and redeploy client configuration

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

This step restarts all services which were modified while enabling HDFS encryption. To complete this step:
  1. Click Restart stale services and redeploy client configuration.
  2. Click Restart Stale Services.
  3. Make sure that Re-deploy client configuration is checked, and click Restart Now.
  4. After all commands have completed, click Finish.

9. Validate Data Encryption

Minimum Required Role: Key Administrator or Cluster Administrator (also provided by Full Administrator)

This step launches a tutorial with instructions on creating an encryption zone and putting data into it to verify that HDFS encryption is enabled and working.

Enabling HDFS Encryption Using a Java KeyStore

After selecting A file-based password-protected Java KeyStore as the root of trust, the following steps are displayed:

1. Enable Kerberos

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

For more information on enabling Kerberos, see Enabling Kerberos Authentication Using the Wizard.

2. Enable TLS/SSL

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

For more information on enabling TLS, see Configuring TLS Security for Cloudera Manager.

3. Add a Java KeyStore KMS Service

Minimum Required Role: Key Administrator (also provided by Full Administrator)

This step adds the Java KeyStore KMS service to the cluster. The Java KeyStore KMS service uses a password-protected Java KeyStore for cryptographic key management. To complete this step:
  1. Click Add a Java KeyStore KMS Service.
  2. Select a cluster host for the Java KeyStore KMS service. Click Continue.
  3. The Setup TLS for Java KeyStore KMS page provides high-level instructions for configuring TLS communication between the EDH cluster and the Java KeyStore KMS. See Configuring TLS/SSL for the KMS for more information.

    Click Continue.

  4. The Review Changes page lists the Java KeyStore settings. Click the icon next to any setting for information about that setting. Enter the location and password for the Java KeyStore and click Continue.
  5. Click Continue to automatically configure the HDFS service to depend on the Java KeyStore KMS service.
  6. Click Finish to complete this step and return to the main page of the wizard.

4. Restart stale services and redeploy client configuration

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

This step restarts all services which were modified while enabling HDFS encryption. To complete this step:
  1. Click Restart stale services and redeploy client configuration.
  2. Click Restart Stale Services.
  3. Make sure that Re-deploy client configuration is checked, and click Restart Now.
  4. After all commands have completed, click Finish.

5. Validate Data Encryption

Minimum Required Role: Key Administrator or Cluster Administrator (also provided by Full Administrator)

This step launches a tutorial with instructions on creating an encryption zone and putting data into it to verify that HDFS encryption is enabled and working.