Enabling Key Trustee KMS High Availability

CDH 5.4.0 and higher supports Key Trustee KMS high availability. For new installations, you can use the Set up HDFS Data At Rest Encryption wizard to install and configure Key Trustee KMS high availability. If you have an existing standalone Key Trustee KMS service, use the following procedure to enable Key Trustee KMS high availability:

Back up the Key Trustee KMS private key and configuration directory. See Backing Up and Restoring Key Trustee Server and Clients for more information.
If you do not have a ZooKeeper service in your cluster, add one using the instructions in Adding a Service.
Run the Add Role Instances wizard for the Key Trustee KMS service (Key Trustee KMS service > Actions > Add Role Instances).
Click Select hosts and check the box for the host where you want to add the additional Key Management Server Proxy role. See Resource Planning for Data at Rest Encryption for considerations when selecting a host. Click OK and then Continue.
On the Review Changes page of the wizard, confirm the authorization code, organization name, and settings, and then click Finish.
Go to Key Trustee KMS service > Configuration and make sure that the ZooKeeper Service dependency is set to the ZooKeeper service for your cluster.
Restart the Key Trustee KMS service (Key Trustee KMS service > Actions > Restart).
Synchronize the Key Trustee KMS private key.
Warning: It is very important that you perform this step. Failure to do so leaves Key Trustee KMS in a state where keys are intermittently inaccessible, depending on which Key Trustee KMS host a client interacts with, because cryptographic key material encrypted by one Key Trustee KMS host cannot be decrypted by another. If you are already running multiple Key Trustee KMS hosts with different private keys, immediately back up all Key Trustee KMS hosts, and contact Cloudera Support for assistance correcting the issue.
To determine whether the Key Trustee KMS private keys are different, compare the MD5 hash of the private keys. On each Key Trustee KMS host, run the following command:
```
$ md5sum /var/lib/kms-keytrustee/keytrustee/.keytrustee/secring.gpg
```
If the outputs are different, contact Cloudera Support for assistance. Do not attempt to synchronize existing keys. If you overwrite the private key and do not have a backup, any keys encrypted by that private key are permanently inaccessible, and any data encrypted by those keys is permanently irretrievable. If you are configuring Key Trustee KMS high availability for the first time, continue synchronizing the private keys.
Cloudera recommends following security best practices and transferring the private key using offline media, such as a removable USB drive. For convenience (for example, in a development or testing environment where maximum security is not required), you can copy the private key over the network by running the following rsync command on the original Key Trustee KMS host:
```
rsync -zav /var/lib/kms-keytrustee/keytrustee/.keytrustee root@ktkms02.example.com:/var/lib/kms-keytrustee/keytrustee/.
```
Replace ktkms02.example.com with the hostname of the Key Trustee KMS host that you are adding.
Restart the cluster.
Redeploy the client configuration (Home > Cluster-wide > Deploy Client Configuration).
Re-run the steps in Validating Hadoop Key Operations.

Cloudera Navigator Key Trustee Server High Availability

Enabling Navigator HSM KMS High Availability