Navigator Encrypt key migration with HSM

Register Navigator Encrypt and start Navigator Encrypt service.

Register the Navigator Encrypt with KTS by executing the following command:
```
 sudo navencrypt register --server=https://<cluster-name>-4.vpc.cloudera.com:11371   --passive-server=https://<cluster-name>-5.vpc.cloudera.com:11371 --org=qa-test --auth=cloudera --skip-ssl-check → Add passphrase and register active and passive servers. 
```
note
After this is working, you can start Navigator Encrypt and start encrypting. You can encrypt two types of devices; a Linux loop device and a physical device. This topic describes how to use a loop device.
Start Navigator Encrypt service by executing the following command:
```
sudo systemctl start navencrypt-mount
```
View unused loop devices by executing the following command:
```
sudo losetup -f
```
You can then use the unused loop devices found by the above command.
Create a new directory called navencrypt_loop_devices by executing the following command:
```
sudo mkdir /navencrypt_loop_devices
```
Datastore files for loop devices go here.
Create loop0, which becomes one of the mount points, inside nav mount:
```
sudo mkdir -p /navencrypt_mount/loop0
```
Create an input file called diskimageloop0 of customized size for the unused loop device found in step c:
```
sudo dd if=/dev/zero of=/navencrypt_loop_devices/diskimageloop0 bs=100M count=10 
```
The dd command creates a 500 GB file. You need to modify the bs and count values to generate the required file size.
Use the available loop device with the navencrypt-prepare -d command to prepare the loop device mount point:
```
sudo navencrypt-prepare  -d  /navencrypt_loop_devices/diskimageloop0 /dev/loop0 /navencrypt_mount/loop0
```
Execute the following command to get hostnames:
```
sudo yum install jq
```
Run ssh using the keyboard-interactive password authentication mode:
```
sudo yum install sshpass
```

Install upgraded rpms for ctrustee, libkeytrustee, navencrypt-kernel-module, navencrypt from successful GBN with the following commands:

sudo yum install ctrustee-7.3.1.0-8_el9-8.x86_64.rpm<add latest rpm>
sudo yum install libkeytrustee-7.3.1.0-8_el9-8.x86_64.rpm
sudo yum install navencrypt-kernel-module-7.3.1.0-8_el9.x86_64.rpm
sudo yum install navencrypt-7.3.1.0-8_el9.x86_64.rpm

Ensure that the uuids.sh and generate_deposits.sh files are present in the /etc/navencrypt folder.

Ensure that ztrustee is present in the /usr/bin/ folder.

If not, try to install ctrustee.rpm again.

Generate the gpg key.

[root@nekts-mhsm-rhel88-1 navencrypt]# gpg --list-secret-keys
gpg: checking the trustdb
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
/root/.gnupg/pubring.kbx
------------------------
sec   rsa2048 2024-08-23 [SC]
      AA1F26481634C37682A063B4071E6471A0B8EE9D
uid           [ultimate] Snehalatha Venkatesh <snehalatha.venkatesh@cloudera.com>
ssb   rsa2048 2024-08-23 [E]

[root@nekts-mhsm-rhel88-1 navencrypt]# gpg --full-generate-key
gpg (GnuPG) 2.2.20; Copyright (C) 2020 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
  (14) Existing key from card
Your selection? 1
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 
Requested keysize is 2048 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 0
Key does not expire at all
Is this correct? (y/N) y

GnuPG needs to construct a user ID to identify your key.

Real name: <Name>
Email address: <cloudera email address>
Comment: 
You selected this USER-ID:
    "<Name><email address>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: key 7A234E85BC56BBD0 marked as ultimately trusted
gpg: revocation certificate stored as '/root/.gnupg/openpgp-revocs.d/CB5270B06F39E4BB92BC80017A234E85BC56BBD0.rev'
public and secret key created and signed.

pub   rsa2048 2024-09-10 [SC]
      CB5270B06F39E4BB92BC80017A234E85BC56BBD0
uid                      Snehalatha Venkatesh <snehalatha.venkatesh@cloudera.com>
sub   rsa2048 2024-09-10 [E]

To hold the gpg key for further purpose, execute the following lines:

echo “gpg_passphrase” > /etc/navencrypt/passphrase.txt -> “gpg_passphrase” is the passphrase given while generating gpg key
Example: echo “cloudera123” > /etc/navencrypt/passphrase.txt

chmod 600 /etc/navencrypt/passphrase.txt

Run the uuids.sh shell script and ensure that you get the success message and check for the reterived_keys file in the /etc/navencrypt folder, as follows:

[root@nekts-keyhsm-1 navencrypt]# sh uuids.sh 
Processing UUID: yMex1OQ5Nb2g674GP7wOiTmevXL0GSAlXZMSGU5ZRSq
Successfully retrieved and saved key for UUID yMex1OQ5Nb2g674GP7wOiTmevXL0GSAlXZMSGU5ZRSq.

Run the generate_deposits.sh file and get the success message, as follows, and check for deposits.csv and consolidated_deposits.csv files in the /etc/navencrypt folder.

[root@tgnekts-mhsm-rhel8-1 navencrypt]# sh generate_deposits.sh -> Enter passphrase given for gpg key.
Key file content encrypted and signed successfully.
Added UUID 31D77qRTgZBe5HkgAdp78uONhJhOI1Jh0hdqpXrKLCB to /etc/navencrypt/deposits.csv.
Deposits CSV file created at /etc/navencrypt/deposits.csv.
Enter Cloudera Manager hostname: tgnekts-mhsm-rhel8-1.vpc.cloudera.com
Enter Cloudera Manager port: 7183
Node list: tgnekts-mhsm-rhel8-1.vpc.cloudera.com,tgnekts-mhsm-rhel8-2.vpc.cloudera.com,tgnekts-mhsm-rhel8-3.vpc.cloudera.com,tgnekts-mhsm-rhel8-4.vpc.cloudera.com,tgnekts-mhsm-rhel8-5.vpc.cloudera.com
Ensuring SSH access for tgnekts-mhsm-rhel8-1.vpc.cloudera.com...
Generating public/private rsa key pair.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:iL/yH/5sz98T4pFZYmzQ4xYgegjliElK5F5Fqu9EPpU root@tgnekts-mhsm-rhel8-1.vpc.cloudera.com
The key's randomart image is:
+---[RSA 3072]----+
|.o ..+.. . .o    |
|o o = + o .. +   |
| o = . + .  + o  |
|. o  ....    B . |
| o ..E. S   + =  |
|  + ..       = . |
|   =  . .   . o .|
|  o o  o o.. . o |
|   . oo.ooo.o.. o|
+----[SHA256]-----+
# tgnekts-mhsm-rhel8-1.vpc.cloudera.com:22 SSH-2.0-OpenSSH_8.0
# tgnekts-mhsm-rhel8-1.vpc.cloudera.com:22 SSH-2.0-OpenSSH_8.0
# tgnekts-mhsm-rhel8-1.vpc.cloudera.com:22 SSH-2.0-OpenSSH_8.0
root@tgnekts-mhsm-rhel8-1.vpc.cloudera.com's password: 
Ensuring SSH access for tgnekts-mhsm-rhel8-2.vpc.cloudera.com...
# tgnekts-mhsm-rhel8-2.vpc.cloudera.com:22 SSH-2.0-OpenSSH_8.0
# tgnekts-mhsm-rhel8-2.vpc.cloudera.com:22 SSH-2.0-OpenSSH_8.0
# tgnekts-mhsm-rhel8-2.vpc.cloudera.com:22 SSH-2.0-OpenSSH_8.0
root@tgnekts-mhsm-rhel8-2.vpc.cloudera.com's password: 
Ensuring SSH access for tgnekts-mhsm-rhel8-3.vpc.cloudera.com...
# tgnekts-mhsm-rhel8-3.vpc.cloudera.com:22 SSH-2.0-OpenSSH_8.0
# tgnekts-mhsm-rhel8-3.vpc.cloudera.com:22 SSH-2.0-OpenSSH_8.0
# tgnekts-mhsm-rhel8-3.vpc.cloudera.com:22 SSH-2.0-OpenSSH_8.0
root@tgnekts-mhsm-rhel8-3.vpc.cloudera.com's password: 
Ensuring SSH access for tgnekts-mhsm-rhel8-4.vpc.cloudera.com...
# tgnekts-mhsm-rhel8-4.vpc.cloudera.com:22 SSH-2.0-OpenSSH_8.0
# tgnekts-mhsm-rhel8-4.vpc.cloudera.com:22 SSH-2.0-OpenSSH_8.0
# tgnekts-mhsm-rhel8-4.vpc.cloudera.com:22 SSH-2.0-OpenSSH_8.0
root@tgnekts-mhsm-rhel8-4.vpc.cloudera.com's password: 
Ensuring SSH access for tgnekts-mhsm-rhel8-5.vpc.cloudera.com...
# tgnekts-mhsm-rhel8-5.vpc.cloudera.com:22 SSH-2.0-OpenSSH_8.0
# tgnekts-mhsm-rhel8-5.vpc.cloudera.com:22 SSH-2.0-OpenSSH_8.0
# tgnekts-mhsm-rhel8-5.vpc.cloudera.com:22 SSH-2.0-OpenSSH_8.0
root@tgnekts-mhsm-rhel8-5.vpc.cloudera.com's password: 
Copying deposits.csv from tgnekts-mhsm-rhel8-1.vpc.cloudera.com...
deposits.csv                                                                                                                                100%  773   903.3KB/s   00:00    
Copying deposits.csv from tgnekts-mhsm-rhel8-2.vpc.cloudera.com...
deposits.csv                                                                                                                                100%  773   903.3KB/s   00:00   
Copying deposits.csv from tgnekts-mhsm-rhel8-3.vpc.cloudera.com...
deposits.csv                                                                                                                                100%  773   903.3KB/s   00:00   
Copying deposits.csv from tgnekts-mhsm-rhel8-4.vpc.cloudera.com...
deposits.csv                                                                                                                                100%  773   903.3KB/s   00:00   
Copying deposits.csv from tgnekts-mhsm-rhel8-5.vpc.cloudera.com...
deposits.csv                                                                                                                                100%  773   903.3KB/s   00:00   
Consolidated CSV file created at /etc/navencrypt/consolidated_deposits.csv.

After the deposits.csv file is created, login to Cloudera Manager UI.

Check for node(instances) in Ranger KMS KTS.

Go to Cloudera Manager > Ranger KMS KTS service.
Go to Instances.
Check the node(instances).

Stop HDFS and Ranger KMS KTS services.

Delete Ranger KMS KTS service from Cloudera Manager UI.

Add Ranger KMS service from Cloudera Manager UI.

To add database details, ssh to Ranger KMS node as shown in step 7 and execute the following commands, where <fqdn> is the node(instance) name:

mysql -u root -p → Login to database with password
CREATE DATABASE rangerkms;
CREATE USER 'rangerkms'@'%' IDENTIFIED BY 'cloudera';
CREATE USER 'rangerkms'@'localhost' IDENTIFIED BY 'cloudera';
CREATE USER 'rangerkms'@'<fqdn>' IDENTIFIED BY 'cloudera';
GRANT ALL PRIVILEGES ON rangerkms.* TO 'rangerkms'@'%';
GRANT ALL PRIVILEGES ON rangerkms.* TO 'rangerkms'@'<fqdn>';
FLUSH PRIVILEGES;

Enable the migration flag under configuration for Ranger KMS and complete the wizard.

Configure the Key password (default value : mykeypass) and Store password (default value : mystorepass) which are used or entered before migration within Ranger KMS KTS service.

Copy the deposits.csv file created in step 5 to the KMS node.

[root@nekts-mig-hsm-3 ~]# scp root@nekts-mig-hsm-1.vpc.cloudera.com:/etc/navencrypt/consolidated_deposits.csv /var/lib/kms-keytrustee/deposits.csv
root@nekts-mig-hsm-1.vpc.cloudera.com's password: 
deposits.csv                                                                                                                100%  664     1.9MB/s   00:00    
[root@nekts-mig-hsm-3 ~]# ls /var/lib/kms-keytrustee/
deposits.csv  keytrustee  kms_bak_nekts-mig-hsm-3_vpc_cloudera_com_2024-08-01_01-25-04.tar  kt_bak_nekts-mig-hsm-3_vpc_cloudera_com_2024-08-01_01-25-04.log

Start the Ranger KMS service.

Create a new file called importPayload.json:

vi /tmp/importPayload.json

Add the following line to the new file called importPayload.json:

{
  "path": "/var/lib/kms-keytrustee/deposits.csv"
}

Run the following API to import Navigator Encrypt keys from the /tmp/deposits.csv file to Ranger KMS DB.

[root@nekts-mig-hsm-3 ~]# kinit systest
Password for systest@VPC.CLOUDERA.COM: 

[root@nekts-mig-hsm-3 ~]# curl -ivk --negotiate -u : -H "Content-Type: application/json" -X POST "https://nekts-mig-hsm-3.vpc.cloudera.com:9494/kms/v2/deposit/import" -d @/tmp/importPayload.json