Known Issues in Cloudera Navigator 6.1.0 Encryption

Known Issues in Cloudera Navigator Key Trustee Server 6.1.0

Additional Key Trustee Server process appears after key creation when passive database is stopped

If a key is created while the passive Key Trustee Server database is down, the key creation will fail with the message, "Database write timed out." In this case, even after the passive database comes up, a hanging Key Trustee Server process thread may remain on the system attempting to complete the write. To view which Key Trustee Server processes are running, enter:
ps -ef | grep keytrustee | grep -v postgres
In normal operations there should be only one Key Trustee Server process running.

Affected Version: 6.0.0, 6.1.0

Cloudera Bug: KT-6684

Workaround: These extra Key Trustee Server processes do not cause any errors with the Key Trustee Server, but they will not be cleaned up properly when the Key Trustee Server is shut down. To fully shut down the Key Trustee Server when there are extra processes running, stop the Key Trustee Server service, and then kill any remaining processes.

Key Trustee Server active database setup fails when command line configuration specifies non-default database port

If the Key Trustee Server is configured from the command line to use a non-default database port (the default port is 11381), then when the Key Trustee Server service is added to Cloudera Manager, the first database startup fails.

Affected Version: 6.0.0, 6.1.0

Cloudera Bug: KT-6238

Workaround:
  1. Log in to the Key Trustee Server and manually stop/start the databases:
    service keytrustee-db stop
    Make sure that the postmaster.pid file, which is located in the database directory, no longer exists. If necessary, replace /var/lib/keytrustee/db in the following command with the appropriate database directory for your system:
    # ls /var/lib/keytrustee/db/postmaster.pid
    If postmaster.pid has not been cleaned up, then use the pg_ctl utility to stop the database directly:
    # pg_ctl stop
  2. Return to the Cloudera Manager home page where the Key Trustee Server service is listed.
  3. Go into the Key Trustee Server service configuration and for Key Trustee Server Database Port, specify the port that was configured during the command line configuration.
  4. Redeploy the Key Trustee Server configuration and restart the service.

Key Trustee KMS cannot connect to Key Trustee Server using TLS versions other than 1.0 on JDK 7

If you have configured Key Trustee Server to use a TLS version other than 1.0, Key Trustee KMS fails to connect to Key Trustee Server, and key operations fail when using JDK 7.

Workaround: Use TLS version 1.0 only, or JDK 8.

Key Trustee Server cannot use TLS version 1.2 on RHEL 6

Configuring Key Trustee Server to use TLS version 1.2 causes Key Trustee Server to be unable to start.

Workaround: Use your operating system package manager to upgrade the pyOpenSSL package to version 1.4 or higher, or do not configure Key Trustee Server to use TLS version 1.2.

Key Trustee Server PKCS8 private key cannot communicate with Key HSM

If its private key is in PKCS8 format, Key Trustee Server cannot communicate with Key HSM.

Affected Version: 6.0.0, 6.1.0

Cloudera Bug: KT-3172

Workaround: Convert the Key Trustee Server private key to raw RSA format.

Known Issues in Cloudera Navigator Key HSM 6.1.0

Roll key command throws an exception and cannot retrieve metadata for key

When using Key Trustee KMS with Key Trustee Server and Key HSM (backed by an HSM device), if there is significant (> 15 ms ping time) network latency between the Key Trustee Server and the HSM device, then EDEK generation errors can occur during the roll key operation. These errors manifest in the KMS log as errors on the generateEncryptedKey operation. The KMS will recover from these errors on its own, but they may represent a nuisance to the operator.

Affected Version: 6.0.0, 6.1.0

Cloudera Bug: KT-5646

Workaround: When these errors occur, you can use the hadoop key list -metadata command to confirm whether or not the key roll was successful, despite the error condition.

KeySecure HSM not supported

The KeySecure HSM is not supported in this release.

Affected Version: 6.0.0, 6.1.0

Cloudera Bug: KT-6001

Workaround: None.

Too many keys on Luna HSM causes Key HSM startup to fail

If there are too many keys on the Luna HSM, Key HSM startup will fail with a Java core dump because it times out querying the keys.

Affected Version: 6.0.0

Cloudera Bug: KT-6129

Fixed in Version: 6.1.0

Workaround: Increase the value in the keyhsm.countdown.time property in application.properties (the default is 45 seconds) until Key HSM starts successfully.

Key HSM Luna setup not showing the correct login status

When running the keyhsm setup luna command, you are prompted for the Luna HSM slot number and login password. Key HSM then attempts to log into the Luna HSM to verify these settings are correct. In some circumstances, the setup script reports that the login was successful, even if it failed.

Affected Version: 6.0.0, 6.1.0

Cloudera Bug: KT-6623

Workaround: Any incorrect settings will cause the Key HSM service to throw an exception upon startup and exit. If the Key HSM service does not start correctly, check the log for the message: "Unable to sign into Luna HSM server. Please rerun application with 'setup' option to configure password." If this message appears, re-run the keyhsm setup luna command, and enter the correct slot number and login password.

The keyhsm trust command fails

When performing setup of Key HSM, the ktadmin keyhsm --server http://server:port --trust command may fail with the following message: "Unable to connect to Key HSM server at this address. Is the server running, and is this KeyTrustee instance trusted (by running keyhsm trust)?" This failure can occur even if you have already successfully executed the keyhsm trust command.

Affected Version: 6.0.0, 6.1.0

Cloudera Bug: KT-6131

Workaround: Re-execute the keyhsm trust command, and then retry the ktadmin keyhsm command.

Upgrading Key HSM removes init script and binary

Upgrading Key HSM from 1.4.x to 1.5.x and higher removes the Key HSM init script and /usr/bin/keyhsm binary.

Workaround: Reinstall Key HSM:
sudo yum reinstall keytrustee-keyhsm

Key HSM cannot trust Key Trustee Server certificate if it has extended attributes

Key HSM cannot trust the Key Trustee Server certificate if it has extended attributes, and therefore cannot integrate with Key Trustee Server.

Workaround: Import the Key Trustee Server certificate to the Key HSM trust store using Java keytool instead of the keyhsm trust command.

Known Issues in Cloudera Navigator Key Trustee KMS 6.1.0

CDH upgrade failure

When upgrading to Key Trustee KMS 6.0.0 from Key Trustee KMS 5.14.0 or lower, and performing a rolling restart (instead of a full restart), the first Key Trustee KMS instance to restart may fail to come up and present the error: "Unable to verify private key match between KMS hosts. If the system has been recently upgraded, DO NOT TAKE FURTHER ACTION and contact your support representative as soon as possible. If this is a new installation, verify private key files have been synched between all KMS hosts. Aborting to prevent data inconsistency."

Affected Versions: 6.0.0

Cloudera Bug: KT-6547

Fixed Version: 6.1.0

Workaround: If possible, perform a full restart instead of a rolling restart.

If you cannot execute a full restart, then add the following line to the /var/lib/kms-keytrustee/keytrustee/.keytrustee/keytrustee.conf file on all Key Trustee KMS instances, and then restart the Key Trustee KMS that failed:
"FINGERPRINT_VALIDATED": "True"

New Key Trustee KMS may fail after being added to an environment that previously had a single Key Trustee KMS instance

When adding a new Navigator Key Trustee KMS instance to an environment that previously only had a single Key Trustee KMS instance, the new Key Trustee KMS may fail to start and return the following message:

"Unable to verify private key match between KMS hosts. If the system has been recently upgraded, DO NOT TAKE FURTHER ACTION and contact your support representative as soon as possible. If this is a new installation, verify private key files have been synced between all KMS hosts. Aborting to prevent data inconsistency."

This message correctly appears if GPG private keys have not been synced across all Key Trustee KMS instances. However, it may erroneously appear when the GPG private keys have been synced, but the original Key Trustee KMS instance has not been restarted to pick up the new configuration.

If you encounter this error message and have either recently upgraded the Key Trustee KMS, or have installed a new Key Trustee KMS service running in HA mode, refer to Key Trustee KMS Encryption Issues for troubleshooting steps.

Affected Version: 6.0.0

Fixed Version: 6.1.0

Cloudera Bug: KT-6231

Workaround: Restart the original Key Trustee KMS instance, and then start the new Key Trustee KMS instance.

The Key Trustee KMS service fails to start if the Trust Store is configured without also configuring the Keystore

If you configure the Key Trustee KMS service Key Management Server Proxy TLS/SSL Certificate Trust Store File and Key Management Server Proxy TLS/SSL Certificate Trust Store Password parameters without also configuring the Key Management Server Proxy TLS/SSL Server JKS Keystore File Location and Key Management Server Proxy TLS/SSL Server JKS Keystore File Password parameters, the Key Trustee KMS service does not start.

Workaround: Configure all Trust Store and Keystore parameters.

Key Trustee KMS backup script fails if PostgreSQL versions lower than 9.3 are installed

If PostgreSQL versions lower than 9.3 are installed on the Key Trustee KMS host, the ktbackup.sh script fails with an error similar to the following:

pg_dump: server version: 9.3.11; pg_dump version: 9.2.14
pg_dump: aborting because of server version mismatch 

Workaround: Uninstall the lower PostgreSQL version.

Known Issues in Cloudera Navigator HSM KMS 6.1.0

Encryption zone key is not deleted after migrating from Key Trustee KMS to HSM KMS

After migrating keys from the Key Trustee KMS to the HSM KMS, the HSM KMS service should be restarted. Until it is restarted, any keys that are deleted after the migration may still be cached. If a deleted key is cached, then the data encrypted with that key can still be accessed even though the key has been deleted.

Affected Version: 6.0.0, 6.1.0

Cloudera Bug: KT-6434

Workaround: Restart the HSM KMS service after the key migration is complete.

KT KMS migration to HSM KMS fails if a key is not found on HSM

When using Key Trustee KMS with Key Trustee Server and Key HSM (backed by an HSM device), if a key version is deleted on the HSM device directly without also deleting and purging that key version on the Key Trustee KMS, then attempts to migrate from the KT KMS to the HSM KMS will fail.

Affected Version: 6.0.0, 6.1.0

Cloudera Bug: KT-5671

Workaround: None.

HSM KMS proxy fails to start during key migration

When migrating keys from the Key Trustee KMS to the HSM KMS, the HSM KMS Proxy may fail to start and return a fatal error.

Affected Version: 6.0.0, 6.1.0

Cloudera Bug: KT-6431

Workaround: Navigate to the Additional Java Configuration Options for KMS setting in the HSM KMS configuration, and ensure that the HSM KMS truststore matches the truststore used by the Key Trustee KMS:
-Djavax.net.ssl.trustStore=/path/to/truststore

You can remove this option after completing the migration.

HSM KMS Thales Proxy role fails to start with non-default Thales HSM Server Port

Port 9001 is used by Cloudera Manager services, and it is also the default privileged port for the Thales HSM KMS. If you use port 9001 for the Thales HSM KMS, it will prevent the CM 6.0.0 upgrade from completing successfully.

Affected Version: 6.0.0, 6.1.0

Cloudera Bug: KT-6403

Workaround: If the Thales HSM KMS already exists, then before upgrading to Cloudera Manager 6.0.0, you must change the privileged Thales HSM KMS port; the recommended port is 11501. The non-privileged port default is 9000, and does not need to be changed.

If you are newly installing the Thales HSM KMS on a 6.0.0 system, then you must set the port to a non-default value before adding the HSM KMS backed by Thales service in Cloudera Manager.

To change the privileged port, log into the Thales HSM KMS machine(s), and run the following commands:
# sudo /opt/nfast/bin/config-serverstartup --enable-tcp --enable-privileged-tcp --privport=11501
[server_settings] change successful; you must restart the hardserver for this to take effect
# sudo /opt/nfast/sbin/init.d-ncipher restart
 -- Running shutdown script 90ncsnmpd

 -- Running shutdown script 60raserv

...

'ncsnmpd' server now running

After successfully running the commands, restart the Thales HSM KMS.

Key store key password specified in Cloudera Manager is not properly used by HSM KMS Service

If the Navigator HSM KMS Proxy TLS/SSL Server JKS Keystore File Password and the Navigator HSM KMS Proxy TLS/SSL Server JKS Keystore Key Password are not both specified or differ from each other, then the HSM KMS inter-node handshake fails and HSM KMS High Availability is not configured.

Affected Version: 6.0.0, 6.1.0

Cloudera Bug: KT-6020

Workaround: Use the same password for the keystore file and the keystore key file.

Timeout error during encryption zone key creation

There are situations where the key cache is synchronously populated to capacity during the create encryption zone operation. The expected behavior is that the key cache is synchronously populated only to the low watermark level (the rest of the keys should be created asynchronously).

Affected Version: 6.0.0, 6.1.0

Cloudera Bug: KT-5296

Workaround: On the HSM KMS, in the field HSM KMS Proxy Advanced Configuration Snippet (Safety Valve) for kms-site.xml:
  • hadoop.security.kms.encrypted.key.cache.low.watermark .05
  • hadoop.security.kms.encrypted.key.cache.size 30
On the HDFS, in the field HDFS Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml:
  • hadoop.security.kms.client.encrypted.key.cache.size 30
  • hadoop.security.kms.client.encrypted.key.cache.low-watermark .05

HSM KMS Luna may need to be restarted if inactive for extended period

If Hadoop key operations return com.safenetinc.luna.exception.LunaCryptokiException after the KMS has been running without activity for an extended period time, the Luna session may have been dropped.

Affected Version(s): 6.0.0, 6.1.0

Cloudera Bug: KT-5018

Workaround: Restart the KMS service.

Creating multiple instances of HSM KMS on the same host and port causes an error upon delete

Creating a KMS role instance on a host that previously hosted a KMS role instance in the same role group that had its data directories deleted results in errors when attempting to run Hadoop key delete operations.

Affected Version(s): 6.0.0, 6.1.0

Cloudera Bug: KT-4992

Workaround: This workaround requires the assistance of Cloudera support; request assistance with issue KT-4992

Incorrect status for "Restart stale services" step in HDFS encryption wizard post-service installation

There are times when completion of the HDFS Encryption Wizard does not show an active "Restart stale services and redeploy client configuration" link.

Affected Version: 6.0.0, 6.1.0

Cloudera Bug: KT-4987

Workaround: Refresh the page and the link should become active.

The encryption wizard continues to fail if there is a failure during initial configuration run

The encryption wizard continues to fail if there was a failure during the initial run configuring HSM KMS.

Affected Version: 6.0.0, 6.1.0

Cloudera Bug: KT-4909

Workaround: Open Cloudera Manager in another browser tab, and manually stop the installed KMS by clicking the arrow next to the KMS and selecting Stop. Then retry the installation in the new tab after correcting the cause of the install failure.

Before installing the Thales backed HSM KMS, you must add the KMS user to the nfast group

After installation of the Thales HSM client, and before installing Navigator HSM KMS backed by Thales HSM, you must add the KMS user to the nfast group..

Affected Version: 6.0.0, 6.1.0

Cloudera Bug: KT-4618

Workaround: Run the following command to manually add the KMS user to the nfast group:usermod -a -G nfast kms

Known Issues in Cloudera Navigator Encrypt 6.1.0

Navigator Encrypt cannot create an ACL when Key Trustee Server is down

Navigator Encrypt cannot create new ACLs when the Key Trustee Server is down. Navigator Encrypt will attempt to verify the master key with the Key Trustee Server when adding ACLs, even if the master key should already have been cached on the local system. If it can't communicate with the Key Trustee Server, the add ACL request will fail.

Affected Version: 6.1.0

Cloudera Bug: KT-6390

Workaround: Make sure the Key Trustee Server is running before making modifications to Navigator Encrypt ACLs.

Navigator Encrypt-related packages should be downgraded with the Navigator Encrypt package

The navencrypt downgrade command will only downgrade the navencrypt package. It will not downgrade the associated navencrypt-kernel-module and libkeytrustee packages.

Affected Version: 6.0.0, 6.1.0

Cloudera Bug: KT-6381

Workaround: To fully downgrade Navigator Encrypt, manually downgrade all of the associated Navigator Encrypt packages (in the order listed):
  1. navencrypt
  2. navencrypt-kernel-module
  3. libkeytrustee

Upgrade from Navigator Encrypt 3.x to 6.0.0 does not trigger a navencryptfs recompile

After upgrading from Navigator Encrypt 3.x to Navigator Encrypt 6.0.0, the kernel module may not be rebuilt, even if the upgrade and installation commands indicate that it was built successfully.

Affected Version: 6.0.0

Cloudera Bug: KT-6382

Fixed in Version: 6.1.0

Workaround: If the version currently loaded is not the desired version, use the following commands to update it. In this case, you must manually load the updated kernel module.

To identify which version of the kernel module is being used, look for the currently loaded version in the dmesg output (in this case, the output reveals that kernel module version was not rebuilt):
navencrypt: time=1531428589 level=INFO app="kernel" Cloudera navencryptfs v3.14.0-405 loaded 
For RHEL/UBUNTU:
service navencrypt-mount stop
navencrypt-module-setup
service navencrypt-mount start   
For SLES:
service navencrypt-mount stop
rmmod navencryptfs
modprobe -v navencryptfs
service navencrypt-mount start

When a mount point is added Navigator Encrypt updates configuration files before the action completes

The navencrypt-prepare command sometimes performs updates to the /etc/navencrypt/control and /etc/navencrypt/ztab files before the command has completed. In such cases, if there is an error with the mounting or unmounting of the navencrypt mount point, then the updated control and ztab files will not accurately reflect which mount points exist. Do not manually change these files without having a known working backup.

Affected Version: 6.0.0

Cloudera Bug: KT-6383

Fixed in Version: 6.1.0

Workaround: None.

Issue with navencrypt-mount service status message

Affected Version: 6.0.0, 6.1.0

Cloudera Bug: KT-6309

On RHEL 7, the service navencrypt-mount stop command might return a successful exit status message, even if there was a problem stopping Navigator Encrypt. This is not an issue with the navencrypt-mount service command; rather, it is a problem with the message output.

You can verify that navencrypt-mount stopped successfully by checking for the presence of the navencryptfs module in the lsmod output:
# lsmod | grep navencrypt
navencryptfs 101407 0
Alternatively, you can verify that the navencrypt mount points successfully dismounted by checking the output of the df or mount commands.

Workaround: None.