Key Trustee KMS Encryption Issues

Description: You may see the following error after you attempt to restart a Key Trustee KMS HA host after an upgrade:

java.io.IOException: Unable to verify private key match between KMS hosts. Verify private key files have been synced between all KMS hosts. Aborting to prevent data inconsistency.        

Solution: If you have failed to synchronize private keys between Key Trustee KMS hosts, they may be in a state where keys are intermittently inaccessible, depending on which Key Trustee KMS host a client interacts with, because cryptographic key material encrypted by one Key Trustee KMS host cannot be decrypted by another. If you are already running multiple Key Trustee KMS hosts with different private keys, immediately back up all Key Trustee KMS hosts, and contact Cloudera Support for assistance correcting the issue.

Description: You may see the following error upon the first run after adding a new Key Trustee KMS HA service:

java.io.IOException: Unable to verify private key match between KMS hosts. Verify private key files have been synced between all KMS hosts. Aborting to prevent data inconsistency.         

Solution: See Validating Private Key Synchronization (Key Trustee KMS HA Only) for guidance on synchronization and validation of private keys.

Cloudera recommends following security best practices and transferring the private key using offline media, such as a removable USB drive. For convenience (for example, in a development or testing environment where maximum security is not required), you can copy the private key over the network by running the rsynccommand on the original Key Trustee KMS host:

rsync -zav /var/lib/kms-keytrustee/keytrustee/.keytrustee root@ktkms02.example.com:/var/lib/kms-keytrustee/keytrustee/.

Replace ktkms02.example.com with the host name of the Key Trustee KMS host that you are adding.

Description: You may see the following error after you attempt to restart a Key Trustee KMS for the first time:

java.lang.Exception: Could not establish connection to ZooKeeper to verify KMS host private key consistency. Verify private key files have been synced between all KMS hosts. Aborting to prevent data inconsistency.        

Solution: ZooKeeper is used to communicate with hosts and is also the storage location of private key data, and therefore must be running upon the first restart or running of the GPG validation check, which compares private keys amongst Key Trustee KMS hosts to help prevent a "split brain" scenario (when private keys are not synchronized between hosts). To ensure the GPG validation check can run, start ZooKeeper, and then restart the Key Trustee KMS.