Key migration in UCL

Learn how to migrate keys from Key Trustee Server (KTS) to Ranger KMS if you have already upgraded the Cloudera Manager version and want to upgrade the Cloudera version to 7.3.1 and above.

To upgrade your Cloudera version to 7.3.1 and above, you first need to upgrade to Cloudera Private Cloud Base 7.1.9. After you upgrade to Cloudera Private Cloud Base 7.1.9, you need to migrate your keys from KTS to Ranger KMS. After you migrate your keys, you can upgrade the Cloudera version to 7.3.1 and above from Cloudera Manager.

If you have upgraded your Cloudera Manager version, and proceed to upgrade your Cloudera version to 7.3.1 and above without migrating your keys from KTS to Ranger KMS, the following validation appears on the upgrade wizard page:

<service_name> in cluster <cluster_name> has been discontinued since 7.3.1 release, please follow this Migration doc to migrate your keys and install Ranger KMS to proceed. Note that if you are using an HSM, you cannot proceed with an upgrade at this time.

Please read through and understand the root cause.

Root Cause: Services mentioned in the cluster are no longer supported in Cloudera 7.3.1 and have to be uninstalled before upgrading to the new version. Before uninstalling any services, you must be diligent to migrate any necessary data and keys stored in the respective services. If KTS service is installed in a separate cluster other than the one you intend to upgrade, it is recommended but not compulsory to uninstall this service.

Confirm the presence for keys in HDFS and Navigator Encrypt. Perform the following steps to locate the keys in KTS:

Login to Ranger UI with key admin credentials.
Go to Key Management > Select Service to view the HDFS encryption zone keys with service Ranger KMS KTS.

If you have setup Navigator Encrypt, locate its keys:

SSH into the active KTS node.
Login to Postgres 14.2 database.

Update the keytrustee user in /etc/passwd before accessing the database by running the following command:

sed -i "/keytrustee:x:$( id -u keytrustee ):$( id -g keytrustee ):Keytrustee User:\/var\/lib\/keytrustee:\/sbin\/nologin/c\keytrustee:x:$( id -u keytrustee ):$( id -g keytrustee ):Keytrustee User:\/var\/lib\/keytrustee:\/bin\/bash" /etc/passwd

Run the following commands to locate the keys:

sudo -u keytrustee LD_LIBRARY_PATH=/opt/cloudera/parcels/KEYTRUSTEE_SERVER/PG_DB/opt/postgres/14.2/lib /opt/cloudera/parcels/KEYTRUSTEE_SERVER/PG_DB/opt/postgres/14.2/bin/psql -p 11381 keytrustee
                           
select handle from deposit;
handle  
---------
mykey1
mykey2
                           
control
                           
control
(6 rows)

Take backup of Ranger KMS KTS and KTS services and migrate the keys from Ranger KMS KTS and KTS services.
note
If needed, change the store password and key password before executing the backup command.
- The keystore file is protected using store password (default value: mystorepass) and key password (default value: mykeypass), that are configurable in Ranger KMS KTS configuration. Change the values as per your requirement.
- Record the key password and store password. You need them after migration while importing keys in Ranger KMS DB.
1. Go to Cloudera Manager.
2. Go to the cluster where Ranger KMS KTS is installed.
3. Click Actions > Backup Keys from Keytrustee Server.
  
  The Backup Keys from Keytrustee Server command is a cluster level command.
  - The backup of the KTS directory is created at /var/lib/keytrustee/ on the active KTS node.
  - The Backup Keys from Keytrustee Server command generates the CSV file required to import Navigator Encrypt keys in Ranger KMS DB after migration. The deposits.csv file is created at /var/lib/keytrustee/.keytrustee.
  - The service GPG keys backup and the migratedKeyStore.jceks keystore file are created at /var/lib/kms-keytrustee on the Ranger KMS KTS node.
Verify that the deposits.csv file is generated at /var/lib/keytrustee/.keytrustee/ on the active KTS node, by running the following command:
```
ls -ltr /var/lib/keytrustee/.keytrustee/deposits.csv
```
Verify that the migratedKeystore.jceks file is generated at /var/lib/kms-keytrustee/keytrustee on the Ranger KMS KTS node, by running the following command:
```
ls -lrt /var/lib/kms-keytrustee/keytrustee/migratedKeyStore.jceks
```
Stop the HDFS and Ranger KMS KTS services.
Remove service dependency of Ranger KMS KTS from the HDFS service.
Since Ranger KMS and Ranger KMS KTS services have similar configurations, take backup of the configurations that you modify as per your requirements. You can copy the configurations somewhere.
Here is a list of common configurations which should be migrated to Ranger KMS:
- RANGER_KMS_KTS_service_env_safety_valve
- ranger_kms_max_heap_size
- ranger_kms_kts_client_java_opts
- conf/dbks-site.xml_role_safety_valve
- conf/kms-site.xml_role_safety_valve
- conf/ranger-kms-site.xml_role_safety_valve
- RANGER_KMS_SERVER_KTS_role_env_safety_valve
- Max_log_size
- Process_username
- Process_groupname
- Kerberos_princ_name
- Ranger_accesslog_rotate_max_days
- Zookeeper_service
- Hadoop_kms_authentication_signer_secret_provider_zookeeper_path
- Hadoop_kms_authentication_signer_secret_provider
- Hadoop_kms_authentication_signer_secret_provider_zookeeper_auth_type
note
The above list is not conclusive.
Delete Ranger KMS KTS service from the Cloudera Manager UI.
Stop the KTS service from the Cloudera Manager UI, even if it is installed in another cluster.
Delete KTS service from the Cloudera Manager UI.

note
If the KTS service is installed in a separate cluster, you can ignore this step. If The KTS service is on the same cluster, you must delete it.
Install the Ranger KMS service from the Cloudera Manager UI and follow the steps as per the wizard.

For more info, see related link for Configuring a database for Ranger or Ranger KMS.
note
Install Ranger KMS DB service on the same cluster from where you have deleted Ranger KMS KTS service. Cloudera also recommends using the same hosts where Ranger KMS KTS service was installed.
Enable the Enable Ranger KMS KTS Migration flag and complete the wizard.
1. Go to Cloudera Manager > Ranger KMS > Configuration.
2. Select Enable Ranger KMS KTS Migration.
note
You must keep this property enabled even after migration. Disabling this property impacts the encryption zone key retrieval.
Start the Ranger KMS DB service.

Confirm that the service is successfully started and functional.
To confirm successful start of Ranger KMS service, perform the following steps:
1. Login to Ranger UI with the user having access to Ranger KMS.
2. Go to Service Manager > Edit KMS service repository.
3. Click Test Connection at the bottom of the page.
  
  This establishes a connection to the Ranger KMS database. Confirm if the connection is successful.
4. Got to Audits > Plugins.
5. Confirm if the plugin policies are successfully synced and have a 200 Http Response Code.

Login to the Ranger KMS Server host and copy the deposits.csv and migratedKeyStore.jceks files to /var/lib/kms-keytrustee/keytrustee/.

scp root@<ranger-kms-hostname>:/var/lib/kms-keytrustee/keytrustee/migratedKeyStore.jceks /var/lib/kms-keytrustee/keytrustee/

scp root@<ranger-kms-hostname>:/var/lib/keytrustee/.keytrustee/deposits.csv /var/lib/kms-keytrustee/keytrustee/

ls -lrt /var/lib/kms-keytrustee/keytrustee/deposits.csv
ls -lrt /var/lib/kms-keytrustee/keytrustee/migratedKeyStore.jceks

Run Import Keys from KTS command from Ranger KMS DB.
1. Go to Cloudera Manager > Ranger KMS.
2. Select Actions > Import Keys From KTS.
Confirm the successful execution of Import Keys from KTS command.

A message saying Keys from {keystore_file_path} has been successfully imported into RangerDB appears.
Ensure service dependency of Ranger KMS is selected from HDFS service.
Start HDFS service.

Verify the successful migration of keys from the Ranger UI.

Login to Ranger UI with key admin credentials.
Go to Key Management > Select Service, to view the keys.

For Navigator Encrypt keys, login to Ranger KMS DB and check for Ranger KMS tables.

The Navigator Encrypt keys are visible in Ranger KMS DB.

mysql -u root -p 
                
 use rangerkms;
 
 show tables;
                +---------------------+
                | Tables_in_rangerkms |
                +---------------------+
                | navencrypt_deposit  |
                | ranger_keystore     |
                | ranger_masterkey    |
                +---------------------+
                3 rows in set (0.00 sec)

After successful verification, proceed to upgrade the Cloudera version from Cloudera Manager.

note

Post migration confirmations:

Once migration is completed successfully, navigate back to the upgrade wizard and click on Run All Checks Again. If the migration steps are performed correctly now the validation errors should be resolved and you will be able to proceed for the upgrade.
A confirmation message appears to get user confirmation for a successful migration of KTS keys. Please select the checkbox to proceed.
If you decide not to use encryption and do not want to migrate the keys to Ranger KMS DB, you can uninstall the blocking services and provide your confirmation through the check box.