Migrate a Key Trustee KMS Server Role Instance to a New Host

After ensuring that the requirements for migrating the Key Trustee KMS Server role instance to a new instance are satisfied, initiate the migration process.

Back up the Key Trustee KMS private key and configuration directory.
Before adding the new role instance, see Resource Planning for Data at Rest Encryption for considerations when selecting a host.
Run the Add Role Instances wizard for the Key Trustee KMS service (Key Trustee KMS service > Actions > Add Role Instances).
Click Select hosts and select the checkbox for the host to which you are adding the new Key Management Server proxy service role instance.

Click OK and then Continue.
On the Review Changes page of the wizard, confirm the authorization code, organization name, and settings, and then click Finish.
Select and start the new KMS instance (Actions for Selected > Start).
important
The initial startup of the KMS instance may fail with the following error message:
```
java.io.IOException: Unable to verify private key match between KMS hosts. 
If the system has been recently upgraded, DO NOT TAKE FURTHER ACTION and contact your 
support representative as soon as possible. If this is a new installation, verify 
private key files have been synced between all KMS hosts. Aborting to prevent data inconsistency.
```
If this occurs, it indicates that the KMS attempted to verify the Key Trustee private key has been synchronized with the new instance, but was unable to because that synchronization has not yet taken place. This is expected behavior at this point in the process. Proceed to the next step, and the new KMS instance will come up when the KMS service is restarted after the synchronization.
Verify that a Kerberos HTTP principal has been created for that specific host role instance in the Security configuration interface (Administration > Security > Kerberos Credentials).
For example, in this scenario the existing KMS Kerberos principal is HTTP/ktkms01.example.com@EXAMPLE.COM , and you must verify the new host principal HTTP/ktkms03.example.com@EXAMPLE.COM has been created before continuing. If you cannot verify the new host principal, then click Generate Missing Credentials on the Kerberos Credentials tab to generate it before continuing.

If the custom Kerberos keytab retrieval script is in use for Kerberos integration, it is important to have those keytabs ready and ingested before proceeding.
Synchronize the Key Trustee KMS private key.
In this use case you log into the original working KMS instance host, which is ktkms01.example.com , and synchronize the keys from your existing KMS host ktkms01.example.com to the new host to which you are migrating, ktkms03.example.com. Copy the private key over the network by running the following command as a privileged (root) user on the original Key Trustee KMS host:
```
rsync -zav /var/lib/kms-keytrustee/keytrustee/.keytrustee root@ktkms03.example.com:/var/lib/kms-keytrustee/keytrustee/.
```
Replace the host name (here, it is ktkms03.example.com) with the hostname of the Key Trustee KMS host to which you are migrating.
warning
It is very important that you perform this step. If the KMS private key is not copied from the existing instance to the new instance, then the encryption keys will be inaccessible when the old KMS instance is removed from the cluster.
To verify that the Key Trustee KMS private keys successfully synchronized, compare the MD5 hash of the private keys.
On each Key Trustee KMS host, run the following command:
```
$ md5sum /var/lib/kms-keytrustee/keytrustee/.keytrustee/secring.gpg
```
If the outputs are different, contact Cloudera Support for assistance. Do not attempt to synchronize existing keys. If you overwrite the private key and do not have a backup, any keys encrypted by that private key are permanently inaccessible, and any data encrypted by those keys is permanently irretrievable. If you are configuring Key Trustee KMS high availability for the first time, continue synchronizing the private keys.
Restart the Key Trustee KMS service (Key Trustee KMS service > Actions > Restart).
After the restart completes, click Close.
Restart the cluster.
This refreshes all the KMS instances in the cluster, and ensures all stale services are restarted.
Redeploy the client configuration (Home > > Cluster > > Deploy Client Configuration).
Run the steps in Validating Hadoop Key Operations.
Perform the check multiple times to exercise the Load balanced KMS nodes properly. If a command fails during this test, stop immediately, halt the cluster, and contact Cloudera Support.
Remove the old KMS instance being migrated.
First stop the KMS instance (ktkms01.example.com) (Select KMS instance > Actions for Selected > Stop), and then delete it (Select KMS instance > > Actions for Selected > > Delete).

Delete the /var/lib/kms-keytrustee directories on the old KMS instance only after configuring and verifying the new KMS instances. Delete this material only after completing the following tasks:
1. Remove the old KMS instances from the KMS service.
2. Verify that you can read previously encrypted data only using the new KMS instances.
Restart the cluster.
This refreshes all the KMS instances in the cluster, and ensures all stale services are restarted.
Redeploy the client configuration (Home > Clusterwide > Deploy Client Configuration).
Re-run the steps in Validating Hadoop Key Operations.
Perform the check multiple times to exercise the Load balanced KMS nodes properly. If a command fails during this test, stop immediately, halt the cluster, and contact Cloudera Support.
Repeat these steps for any additional KMS node migrations you wish to perform.
In the use case shown here, repeat the steps to migrate the ktkms02.example.com host to the ktkms04.example.com host.