Cloudera Navigator Key Trustee Server High Availability

Key Trustee Server high availability applies to read operations only. If either Key Trustee Server fails, the KeyProvider automatically retries fetching keys from the functioning server. New write operations (for example, creating new encryption keys) are not allowed unless both Key Trustee Servers are operational.

If a Key Trustee Server fails, the following operations are impacted:

  • HDFS Encryption
    • You cannot create new encryption keys for encryption zones.
    • You can write to and read from existing encryption zones, but you cannot create new zones.
  • Cloudera Navigator Encrypt
    • You cannot register new Cloudera Navigator Encrypt clients.
    • You can continue reading and writing encrypted data, including creating new mount points, using existing clients.

Cloudera recommends monitoring both Key Trustee Servers. If a Key Trustee Server fails catastrophically, restore it from backup to a new host with the same hostname and IP address as the failed host. See Backing Up and Restoring Key Trustee Server and Clients for more information. Cloudera does not support PostgreSQL promotion to convert a passive Key Trustee Server to an active Key Trustee Server.

Depending on your cluster configuration and the security practices in your organization, you might need to restrict the allowed versions of TLS/SSL used by Key Trustee Server. For details, see Specifying TLS/SSL Minimum Allowed Version and Ciphers.

Continue reading:

Configuring Key Trustee Server High Availability Using Cloudera Manager

For new installations, use the Set up HDFS Data At Rest Encryption wizard and follow the instructions in Enabling HDFS Encryption Using the Wizard. When prompted, make sure that the Enable High Availability option is selected.

If you already have a Key Trustee Server service, and want to enable high availability, use the Add Role Instances wizard for the Key Trustee Server service instead to add the Passive Key Trustee Server and Passive Database roles.
After completing the Add Role Instances wizard, the Passive Key Trustee Server and Passive Database roles fail to start. Complete the following manual actions to start these roles:
  1. Stop the Key Trustee Server service (Key Trustee Server service > Actions > Stop).
  2. Run the Set Up Key Trustee Server Database command (Key Trustee Server service > Actions > Set Up Key Trustee Server Database).
  3. Run the following command on the Active Key Trustee Server: 
    $ sudo rsync -zcav --exclude .ssl /var/lib/keytrustee/.keytrustee root@keytrustee02.example.com:/var/lib/keytrustee/.

    Replace keytrustee02.example.com with the hostname of the Passive Key Trustee Server.

  4. Run the following command on the Passive Key Trustee Server: 
    $ sudo ktadmin init
  5. Start the Key Trustee Server service (Key Trustee Server service > Actions > Start).
  6. Enable synchronous replication (Key Trustee Server service > Actions > Setup Enable Synchronous Replication in HA mode).
  7. Restart the Key Trustee Server service (Key Trustee Server service > Actions > Restart).

For parcel-based Key Trustee Server releases 5.8 and higher, Cloudera Manager automatically backs up Key Trustee Server (using the ktbackup.sh script) after adding the Key Trustee Server service. It also schedules automatic backups using cron. For package-based installations, you must manually back up Key Trustee Server and configure a cron job.

Cloudera Manager configures cron to run the backup script hourly. The latest 10 backups are retained in /var/lib/keytrustee in cleartext. For information about using the backup script and configuring the cron job (including how to encrypt backups), see Backing Up Key Trustee Server and Key Trustee KMS Using the ktbackup.sh Script.

Configuring Key Trustee Server High Availability Using the Command Line

Install and configure a second Key Trustee Server following the instructions in Installing Cloudera Navigator Key Trustee Server.

Once you have installed and configured the second Key Trustee Server, initialize the active Key Trustee Server by running the following commands on the active Key Trustee Server host: 
$ sudo ktadmin init --external-address keytrustee01.example.com
$ sudo rsync -zav --exclude .ssl /var/lib/keytrustee/.keytrustee root@keytrustee02.example.com:/var/lib/keytrustee/.
$ sudo ktadmin db --bootstrap --port 11381 --pg-rootdir /var/lib/keytrustee/db --slave keytrustee02.example.com
## For RHEL/CentOS 7, use 'sudo systemctl [stop|start] <service_name>' instead of 'sudo service <service_name> [stop|start]' ##
$ sudo service keytrustee-db stop
$ sudo service keytrustee-db start
$ sudo service keytrusteed start

Replace keytrustee01.example.com with the fully qualified domain name (FQDN) of the active Key Trustee Server. Replace keytrustee02.example.com with the FQDN of the passive Key Trustee Server. Cloudera recommends using the default /var/lib/keytrustee/db directory for the PostgreSQL database.

To use a different port for the database, modify the ktadmin init and ktadmin db commands as follows:

$ sudo ktadmin init --external-address keytrustee01.example.com --db-connect postgresql://localhost:<port>/keytrustee?host=/tmp
$ sudo ktadmin db --bootstrap --port <port> --pg-rootdir /var/lib/keytrustee/db --slave keytrustee02.example.com
If you use a database directory other than /var/lib/keytrustee/db, create or edit the /etc/sysconfig/keytrustee-db file and add the following line: 
ARGS="--pg-rootdir /path/to/db"

The ktadmin init command generates a self-signed certificate that the Key Trustee Server uses for HTTPS communication.

Initialize the passive Key Trustee Server by running the following commands on the passive host: 
$ sudo ktadmin init-slave --master keytrustee01.example.com --pg-rootdir /var/lib/keytrustee/db --no-import-key --no-start
## For RHEL/CentOS 7, use 'sudo systemctl [stop|start] <service_name>' instead of 'sudo service <service_name> [stop|start]' ##
$ sudo service keytrustee-db start
$ sudo ktadmin init --external-address keytrustee02.example.com
$ sudo service keytrusteed start

Replace keytrustee02.example.com with the fully qualified domain name (FQDN) of the passive Key Trustee Server. Replace keytrustee01.example.com with the FQDN of the active Key Trustee Server. Cloudera recommends using the default /var/lib/keytrustee/db directory for the PostgreSQL database.

To use a different port for the database, modify the ktadmin init-slave command as follows:

$ sudo ktadmin init-slave --master keytrustee01.example.com --pg-port <port> --pg-rootdir /var/lib/keytrustee/db --no-import-key --no-start
If you use a database directory other than /var/lib/keytrustee/db, create or edit the /etc/sysconfig/keytrustee-db file and add the following line: 
ARGS="--pg-rootdir /path/to/db"

The ktadmin init-slave command performs an initial database sync by running the pg_basebackup command. The database directory must be empty for this step to work. For information on performing an incremental backup, see the PostgreSQL documentation.

The ktadmin init command generates a self-signed certificate that the Key Trustee Server uses for HTTPS communication. Instructions for using alternate certificates (for example, if you have obtained certificates from a trusted Certificate Authority) are provided later.

Enable Synchronous Replication

Key Trustee Server high availability requires synchronous replication to ensure that all rows in the database are inserted in at least two hosts, which protects against key loss.

To enable synchronous replication, run the following command on the active Key Trustee Server: 
$ sudo ktadmin enable-synchronous-replication --pg-rootdir /var/lib/keytrustee/db

If you modified the default database location, replace /var/lib/keytrustee/db with the modified path.

(Optional) Replace Self-Signed Certificates with CA-Signed Certificates

If you have a CA-signed certificate for Key Trustee Server, see Managing Key Trustee Server Certificates for instructions on how to replace the self-signed certificates.

Recovering a Key Trustee Server

If a Key Trustee Server fails, restore it from backup as soon as possible. If the Key Trustee Server hosts fails completely, make sure that you restore the Key Trustee Server to a new host with the same hostname and IP address as the failed host.

For more information, see Backing Up and Restoring Key Trustee Server and Clients.