Known Issues in Cloudera Navigator 6.1.0 Encryption
The following sections describes known issues in the encryption components of Cloudera Navigator 6.1.0, grouped by component:
Known Issues in Cloudera Navigator Key Trustee Server 6.1.0
Additional Key Trustee Server process appears after key creation when passive database is stopped
ps -ef | grep keytrustee | grep -v postgresIn normal operations there should be only one Key Trustee Server process running.
Affected Version: 6.0.0, 6.1.0
Cloudera Bug: KT-6684
Workaround: These extra Key Trustee Server processes do not cause any errors with the Key Trustee Server, but they will not be cleaned up properly when the Key Trustee Server is shut down. To fully shut down the Key Trustee Server when there are extra processes running, stop the Key Trustee Server service, and then kill any remaining processes.
Key Trustee Server active database setup fails when command line configuration specifies non-default database port
If the Key Trustee Server is configured from the command line to use a non-default database port (the default port is 11381), then when the Key Trustee Server service is added to Cloudera Manager, the first database startup fails.
Affected Version: 6.0.0, 6.1.0
Cloudera Bug: KT-6238
- Log in to the Key Trustee Server and manually stop/start the databases:
service keytrustee-db stop
Make sure that the postmaster.pid file, which is located in the database directory, no longer exists. If necessary, replace /var/lib/keytrustee/db in the following command with the appropriate database directory for your system:# ls /var/lib/keytrustee/db/postmaster.pid
If postmaster.pid has not been cleaned up, then use the pg_ctl utility to stop the database directly:# pg_ctl stop
- Return to the Cloudera Manager home page where the Key Trustee Server service is listed.
- Go into the Key Trustee Server service configuration and for Key Trustee Server Database Port, specify the port that was configured during the command line configuration.
- Redeploy the Key Trustee Server configuration and restart the service.
Key Trustee KMS cannot connect to Key Trustee Server using TLS versions other than 1.0 on JDK 7
If you have configured Key Trustee Server to use a TLS version other than 1.0, Key Trustee KMS fails to connect to Key Trustee Server, and key operations fail when using JDK 7.
Workaround: Use TLS version 1.0 only, or JDK 8.
Key Trustee Server cannot use TLS version 1.2 on RHEL 6
Configuring Key Trustee Server to use TLS version 1.2 causes Key Trustee Server to be unable to start.
Workaround: Use your operating system package manager to upgrade the pyOpenSSL package to version 1.4 or higher, or do not configure Key Trustee Server to use TLS version 1.2.
Known Issues in Cloudera Navigator Key HSM 6.1.0
Roll key command throws an exception and cannot retrieve metadata for key
When using Key Trustee KMS with Key Trustee Server and Key HSM (backed by an HSM device), if there is significant (> 15 ms ping time) network latency between the Key Trustee Server and the HSM device, then EDEK generation errors can occur during the roll key operation. These errors manifest in the KMS log as errors on the generateEncryptedKey operation. The KMS will recover from these errors on its own, but they may represent a nuisance to the operator.
Affected Version: 6.0.0, 6.1.0
Cloudera Bug: KT-5646
Workaround: When these errors occur, you can use the hadoop key list -metadata command to confirm whether or not the key roll was successful, despite the error condition.
KeySecure HSM not supported
The KeySecure HSM is not supported in this release.
Affected Version: 6.0.0, 6.1.0
Cloudera Bug: KT-6001
Workaround: None.
Too many keys on Luna HSM causes Key HSM startup to fail
If there are too many keys on the Luna HSM, Key HSM startup will fail with a Java core dump because it times out querying the keys.
Affected Version: 6.0.0
Cloudera Bug: KT-6129
Fixed in Version: 6.1.0
Workaround: Increase the value in the keyhsm.countdown.time property in application.properties (the default is 45 seconds) until Key HSM starts successfully.
Key HSM Luna setup not showing the correct login status
When running the keyhsm setup luna command, you are prompted for the Luna HSM slot number and login password. Key HSM then attempts to log into the Luna HSM to verify these settings are correct. In some circumstances, the setup script reports that the login was successful, even if it failed.
Affected Version: 6.0.0, 6.1.0
Cloudera Bug: KT-6623
Workaround: Any incorrect settings will cause the Key HSM service to throw an exception upon startup and exit. If the Key HSM service does not start correctly, check the log for the message: "Unable to sign into Luna HSM server. Please rerun application with 'setup' option to configure password." If this message appears, re-run the keyhsm setup luna command, and enter the correct slot number and login password.
The keyhsm trust command fails
When performing setup of Key HSM, the ktadmin keyhsm --server http://server:port --trust command may fail with the following message: "Unable to connect to Key HSM server at this address. Is the server running, and is this KeyTrustee instance trusted (by running keyhsm trust)?" This failure can occur even if you have already successfully executed the keyhsm trust command.
Affected Version: 6.0.0, 6.1.0
Cloudera Bug: KT-6131
Workaround: Re-execute the keyhsm trust command, and then retry the ktadmin keyhsm command.
Upgrading Key HSM removes init script and binary
Upgrading Key HSM from 1.4.x to 1.5.x and higher removes the Key HSM init script and /usr/bin/keyhsm binary.
sudo yum reinstall keytrustee-keyhsm
Key HSM cannot trust Key Trustee Server certificate if it has extended attributes
Key HSM cannot trust the Key Trustee Server certificate if it has extended attributes, and therefore cannot integrate with Key Trustee Server.
Workaround: Import the Key Trustee Server certificate to the Key HSM trust store using Java keytool instead of the keyhsm trust command.
Known Issues in Cloudera Navigator Key Trustee KMS 6.1.0
CDH upgrade failure
When upgrading to Key Trustee KMS 6.0.0 from Key Trustee KMS 5.14.0 or lower, and performing a rolling restart (instead of a full restart), the first Key Trustee KMS instance to restart may fail to come up and present the error: "Unable to verify private key match between KMS hosts. If the system has been recently upgraded, DO NOT TAKE FURTHER ACTION and contact your support representative as soon as possible. If this is a new installation, verify private key files have been synched between all KMS hosts. Aborting to prevent data inconsistency."
Affected Versions: 6.0.0
Cloudera Bug: KT-6547
Fixed Version: 6.1.0
Workaround: If possible, perform a full restart instead of a rolling restart.
"FINGERPRINT_VALIDATED": "True"
New Key Trustee KMS may fail after being added to an environment that previously had a single Key Trustee KMS instance
When adding a new Navigator Key Trustee KMS instance to an environment that previously only had a single Key Trustee KMS instance, the new Key Trustee KMS may fail to start and return the following message:
"Unable to verify private key match between KMS hosts. If the system has been recently upgraded, DO NOT TAKE FURTHER ACTION and contact your support representative as soon as possible. If this is a new installation, verify private key files have been synced between all KMS hosts. Aborting to prevent data inconsistency."
This message correctly appears if GPG private keys have not been synced across all Key Trustee KMS instances. However, it may erroneously appear when the GPG private keys have been synced, but the original Key Trustee KMS instance has not been restarted to pick up the new configuration.If you encounter this error message and have either recently upgraded the Key Trustee KMS, or have installed a new Key Trustee KMS service running in HA mode, refer to Key Trustee KMS Encryption Issues for troubleshooting steps.
Affected Version: 6.0.0
Fixed Version: 6.1.0
Cloudera Bug: KT-6231
Workaround: Restart the original Key Trustee KMS instance, and then start the new Key Trustee KMS instance.
The Key Trustee KMS service fails to start if the Trust Store is configured without also configuring the Keystore
If you configure the Key Trustee KMS service Key Management Server Proxy TLS/SSL Certificate Trust Store File and Key Management Server Proxy TLS/SSL Certificate Trust Store Password parameters without also configuring the Key Management Server Proxy TLS/SSL Server JKS Keystore File Location and Key Management Server Proxy TLS/SSL Server JKS Keystore File Password parameters, the Key Trustee KMS service does not start.
Workaround: Configure all Trust Store and Keystore parameters.
Key Trustee KMS backup script fails if PostgreSQL versions lower than 9.3 are installed
If PostgreSQL versions lower than 9.3 are installed on the Key Trustee KMS host, the ktbackup.sh script fails with an error similar to the following:
pg_dump: server version: 9.3.11; pg_dump version: 9.2.14 pg_dump: aborting because of server version mismatch
Workaround: Uninstall the lower PostgreSQL version.
Known Issues in Cloudera Navigator HSM KMS 6.1.0
Encryption zone key is not deleted after migrating from Key Trustee KMS to HSM KMS
After migrating keys from the Key Trustee KMS to the HSM KMS, the HSM KMS service should be restarted. Until it is restarted, any keys that are deleted after the migration may still be cached. If a deleted key is cached, then the data encrypted with that key can still be accessed even though the key has been deleted.
Affected Version: 6.0.0, 6.1.0
Cloudera Bug: KT-6434
Workaround: Restart the HSM KMS service after the key migration is complete.
KT KMS migration to HSM KMS fails if a key is not found on HSM
When using Key Trustee KMS with Key Trustee Server and Key HSM (backed by an HSM device), if a key version is deleted on the HSM device directly without also deleting and purging that key version on the Key Trustee KMS, then attempts to migrate from the KT KMS to the HSM KMS will fail.
Affected Version: 6.0.0, 6.1.0
Cloudera Bug: KT-5671
Workaround: None.
HSM KMS proxy fails to start during key migration
When migrating keys from the Key Trustee KMS to the HSM KMS, the HSM KMS Proxy may fail to start and return a fatal error.
Affected Version: 6.0.0, 6.1.0
Cloudera Bug: KT-6431
-Djavax.net.ssl.trustStore=/path/to/truststore
You can remove this option after completing the migration.
HSM KMS Thales Proxy role fails to start with non-default Thales HSM Server Port
Port 9001 is used by Cloudera Manager services, and it is also the default privileged port for the Thales HSM KMS. If you use port 9001 for the Thales HSM KMS, it will prevent the CM 6.0.0 upgrade from completing successfully.
Affected Version: 6.0.0, 6.1.0
Cloudera Bug: KT-6403
Workaround: If the Thales HSM KMS already exists, then before upgrading to Cloudera Manager 6.0.0, you must change the privileged Thales HSM KMS port; the recommended port is 11501. The non-privileged port default is 9000, and does not need to be changed.
If you are newly installing the Thales HSM KMS on a 6.0.0 system, then you must set the port to a non-default value before adding the HSM KMS backed by Thales service in Cloudera Manager.
# sudo /opt/nfast/bin/config-serverstartup --enable-tcp --enable-privileged-tcp --privport=11501 [server_settings] change successful; you must restart the hardserver for this to take effect # sudo /opt/nfast/sbin/init.d-ncipher restart -- Running shutdown script 90ncsnmpd -- Running shutdown script 60raserv ... 'ncsnmpd' server now running
After successfully running the commands, restart the Thales HSM KMS.
Key store key password specified in Cloudera Manager is not properly used by HSM KMS Service
If the Navigator HSM KMS Proxy TLS/SSL Server JKS Keystore File Password and the Navigator HSM KMS Proxy TLS/SSL Server JKS Keystore Key Password are not both specified or differ from each other, then the HSM KMS inter-node handshake fails and HSM KMS High Availability is not configured.
Affected Version: 6.0.0, 6.1.0
Cloudera Bug: KT-6020
Workaround: Use the same password for the keystore file and the keystore key file.
Timeout error during encryption zone key creation
There are situations where the key cache is synchronously populated to capacity during the create encryption zone operation. The expected behavior is that the key cache is synchronously populated only to the low watermark level (the rest of the keys should be created asynchronously).
Affected Version: 6.0.0, 6.1.0
Cloudera Bug: KT-5296
- hadoop.security.kms.encrypted.key.cache.low.watermark .05
- hadoop.security.kms.encrypted.key.cache.size 30
- hadoop.security.kms.client.encrypted.key.cache.size 30
- hadoop.security.kms.client.encrypted.key.cache.low-watermark .05
HSM KMS Luna may need to be restarted if inactive for extended period
If Hadoop key operations return com.safenetinc.luna.exception.LunaCryptokiException after the KMS has been running without activity for an extended period time, the Luna session may have been dropped.
Affected Version(s): 6.0.0, 6.1.0
Cloudera Bug: KT-5018
Workaround: Restart the KMS service.
Creating multiple instances of HSM KMS on the same host and port causes an error upon delete
Creating a KMS role instance on a host that previously hosted a KMS role instance in the same role group that had its data directories deleted results in errors when attempting to run Hadoop key delete operations.
Affected Version(s): 6.0.0, 6.1.0Cloudera Bug: KT-4992
Workaround: This workaround requires the assistance of Cloudera support; request assistance with issue KT-4992
Incorrect status for "Restart stale services" step in HDFS encryption wizard post-service installation
There are times when completion of the HDFS Encryption Wizard does not show an active "Restart stale services and redeploy client configuration" link.
Affected Version: 6.0.0, 6.1.0
Cloudera Bug: KT-4987
Workaround: Refresh the page and the link should become active.
The encryption wizard continues to fail if there is a failure during initial configuration run
The encryption wizard continues to fail if there was a failure during the initial run configuring HSM KMS.
Affected Version: 6.0.0, 6.1.0
Cloudera Bug: KT-4909
Workaround: Open Cloudera Manager in another browser tab, and manually stop the installed KMS by clicking the arrow next to the KMS and selecting Stop. Then retry the installation in the new tab after correcting the cause of the install failure.
Before installing the Thales backed HSM KMS, you must add the KMS user to the nfast group
After installation of the Thales HSM client, and before installing Navigator HSM KMS backed by Thales HSM, you must add the KMS user to the nfast group..
Affected Version: 6.0.0, 6.1.0
Cloudera Bug: KT-4618
Workaround: Run the following command to manually add the KMS user to the nfast group:usermod -a -G nfast kms
Known Issues in Cloudera Navigator Encrypt 6.1.0
Navigator Encrypt cannot create an ACL when Key Trustee Server is down
Navigator Encrypt cannot create new ACLs when the Key Trustee Server is down. Navigator Encrypt will attempt to verify the master key with the Key Trustee Server when adding ACLs, even if the master key should already have been cached on the local system. If it can't communicate with the Key Trustee Server, the add ACL request will fail.
Affected Version: 6.1.0
Cloudera Bug: KT-6390
Workaround: Make sure the Key Trustee Server is running before making modifications to Navigator Encrypt ACLs.
Navigator Encrypt-related packages should be downgraded with the Navigator Encrypt package
The navencrypt downgrade command will only downgrade the navencrypt package. It will not downgrade the associated navencrypt-kernel-module and libkeytrustee packages.
Affected Version: 6.0.0, 6.1.0
Cloudera Bug: KT-6381
- navencrypt
- navencrypt-kernel-module
- libkeytrustee
Upgrade from Navigator Encrypt 3.x to 6.0.0 does not trigger a navencryptfs recompile
After upgrading from Navigator Encrypt 3.x to Navigator Encrypt 6.0.0, the kernel module may not be rebuilt, even if the upgrade and installation commands indicate that it was built successfully.
Affected Version: 6.0.0
Cloudera Bug: KT-6382
Fixed in Version: 6.1.0
Workaround: If the version currently loaded is not the desired version, use the following commands to update it. In this case, you must manually load the updated kernel module.
navencrypt: time=1531428589 level=INFO app="kernel" Cloudera navencryptfs v3.14.0-405 loaded
service navencrypt-mount stop navencrypt-module-setup service navencrypt-mount start
service navencrypt-mount stop rmmod navencryptfs modprobe -v navencryptfs service navencrypt-mount start
When a mount point is added Navigator Encrypt updates configuration files before the action completes
The navencrypt-prepare command sometimes performs updates to the /etc/navencrypt/control and /etc/navencrypt/ztab files before the command has completed. In such cases, if there is an error with the mounting or unmounting of the navencrypt mount point, then the updated control and ztab files will not accurately reflect which mount points exist. Do not manually change these files without having a known working backup.
Affected Version: 6.0.0
Cloudera Bug: KT-6383
Fixed in Version: 6.1.0
Workaround: None.
Issue with navencrypt-mount service status message
Affected Version: 6.0.0, 6.1.0
Cloudera Bug: KT-6309
On RHEL 7, the service navencrypt-mount stop command might return a successful exit status message, even if there was a problem stopping Navigator Encrypt. This is not an issue with the navencrypt-mount service command; rather, it is a problem with the message output.
# lsmod | grep navencrypt navencryptfs 101407 0Alternatively, you can verify that the navencrypt mount points successfully dismounted by checking the output of the df or mount commands.
Workaround: None.