Key Trustee KMS Encryption Issues

The following errors and conditions are related to the Key Trustee KMS, and includes possible workarounds for issues that may arise when using Key Trustee KMS.

Key Trustee KMS Fails to Restart After Upgrade (HA Only)

Description: You may see the following error after you attempt to restart a Key Trustee KMS HA host after an upgrade:

java.io.IOException: Unable to verify private key match between KMS hosts. Verify private key files have been synced between all KMS hosts. Aborting to prevent data inconsistency.

Solution: If you have failed to synchronize private keys between Key Trustee KMS hosts, they may be in a state where keys are intermittently inaccessible, depending on which Key Trustee KMS host a client interacts with, because cryptographic key material encrypted by one Key Trustee KMS host cannot be decrypted by another. If you are already running multiple Key Trustee KMS hosts with different private keys, immediately back up (see "Backing Up and Restoring Key Trustee Server and Clients") all Key Trustee KMS hosts, and contact Cloudera Support for assistance correcting the issue.

For information about how to check if you are using different private keys, see "Upgrading Key Trustee KMS".

Key Trustee KMS Fails Upon First Run (HA Only)

Description: You may see the following error upon the first run after adding a new Key Trustee KMS HA service:

java.io.IOException: Unable to verify private key match between KMS hosts. Verify private key files have been synced between all KMS hosts. Aborting to prevent data inconsistency.

Solution: See "Upgrading Key Trustee KMS" for guidance on synchronization and validation of private keys.

Cloudera recommends following security best practices and transferring the private key using offline media, such as a removable USB drive. For convenience (for example, in a development or testing environment where maximum security is not required), you can copy the private key over the network by running the rsynccommand on the original Key Trustee KMS host:

rsync -zav /var/lib/kms-keytrustee/keytrustee/.keytrustee root@ktkms02.example.com:/var/lib/kms-keytrustee/keytrustee/.

Replace ktkms02.example.com with the host name of the Key Trustee KMS host that you are adding.

Key Trustee KMS Fails to Start Because ZooKeeper is Not Running

Description: You may see the following error after you attempt to restart a Key Trustee KMS for the first time:

java.lang.Exception: Could not establish connection to ZooKeeper to verify KMS host private key consistency. Verify private key files have been synced between all KMS hosts. Aborting to prevent data inconsistency.

Solution: ZooKeeper is used to communicate with hosts and is also the storage location of private key data, and therefore must be running upon the first restart or running of the GPG validation check, which compares private keys amongst Key Trustee KMS hosts to help prevent a "split brain" scenario (when private keys are not synchronized between hosts). To ensure the GPG validation check can run, start ZooKeeper, and then restart the Key Trustee KMS.