Cloudera Navigator Key Trustee Server High Availability

Configuring Key Trustee Server High Availability Using Cloudera Manager

For new installations, add the Key Trustee Server service to your cluster, following the instructions in Adding a Service. When customizing role assignments, assign the Active Key Trustee Server, Active Database, Passive Key Trustee Server, and Passive Database roles.

If you already have a Key Trustee Server service, and want to enable high availability, use the Add Role Instances wizard for the Key Trustee Server service instead to add the Passive Key Trustee Server and Passive Database roles.

The remaining instructions apply to both new installations and adding a passive Key Trustee Server.

After completing the Add a Service or Add Role Instances wizard, the Passive Key Trustee Server and Passive Database roles fail to start. Complete the following manual actions to start these roles:
  • Stop the Key Trustee Server service (Key Trustee Server service > Actions > Stop).
  • Run the Set Up Key Trustee Server Database command (Key Trustee Server service > Actions > Set Up Key Trustee Server Database).
  • Run the following command on the Active Key Trustee Server: 
    $ sudo rsync -zav --exclude .ssl /var/lib/keytrustee/.keytrustee root@keytrustee02.example.com:/var/lib/keytrustee/.

    Replace keytrustee02.example.com with the hostname of the Passive Key Trustee Server.

  • Run the following command on the Passive Key Trustee Server: 
    $ sudo -u keytrustee /opt/cloudera/parcels/KEYTRUSTEE_SERVER/bin/ktadmin init
  • Start the Key Trustee Server service (Key Trustee Server service > Actions > Start).

Enable Synchronous Replication

Key Trustee Server high availability requires synchronous replication to ensure that all rows in the database are inserted in at least two hosts, protecting against key loss.

To enable synchronous replication, run the following command on the active Key Trustee Server: 
$ sudo -u keytrustee ktadmin enable-synchronous-replication --pg-rootdir /var/lib/keytrustee/db

If you modified the default database location, replace /var/lib/keytrustee/db with the modified path.

Configuring Key Trustee Server High Availability Using the Command Line

Install and configure a second Key Trustee Server following the instructions in Installing Cloudera Navigator Key Trustee Server.

Once you have installed and configured the second Key Trustee Server, initialize the active Key Trustee Server by running the following commands on the active Key Trustee Server host: 
$ sudo -u keytrustee ktadmin init --logdir /var/log/keytrustee --external-address keytrustee01.example.com
$ sudo rsync -zav --exclude .ssl /var/lib/keytrustee/.keytrustee root@keytrustee02.example.com:/var/lib/keytrustee/.
$ sudo -u keytrustee ktadmin db --bootstrap --port 11381 --pg-rootdir /var/lib/keytrustee/db --slave keytrustee02.example.com
$ sudo /etc/init.d/keytrusteed start

Replace keytrustee01.example.com with the fully-qualified domain name (FQDN) of the active Key Trustee Server, keytrustee02.example.com with the FQDN of the passive Key Trustee Server, and /var/lib/keytrustee/db with the path to the directory you want to use to store the PostgreSQL database..

The ktadmin init command generates a self-signed certificate that the Key Trustee Server uses for HTTPS communication.

Initialize the passive Key Trustee Server by running the following commands on the passive host: 
$ sudo -u keytrustee ktadmin init-slave --master keytrustee01.example.com --pg-rootdir /var/lib/keytrustee/db --no-import-key --logdir /var/lib/keytrustee/.keytrustee/logs --no-start
$ sudo -u keytrustee /usr/pgsql-9.3/bin/pg_ctl -D /var/lib/keytrustee/db start
$ sudo -u keytrustee ktadmin init --external-address keytrustee02.example.com
$ sudo /etc/init.d/keytrusteed start

Replace keytrustee02.example.com with the fully-qualified domain name (FQDN) of the passive Key Trustee Server, keytrustee01.example.com with the FQDN of the active Key Trustee Server, and /var/lib/keytrustee/db with the path to the directory you want to use to store the PostgreSQL database.

The ktadmin init-slave command performs an initial database sync by running the pg_basebackup command. The database directory must be empty for this step to work. For information on performing an incremental backup, see the PostgreSQL documentation.

The ktadmin init command generates a self-signed certificate that the Key Trustee Server uses for HTTPS communication. Instructions for using alternate certificates (for example, if you have obtained certificates from a trusted Certificate Authority) are provided later.

Enable Synchronous Replication

Key Trustee Server high availability requires synchronous replication to ensure that all rows in the database are inserted in at least two hosts, protecting against key loss.

To enable synchronous replication, run the following command on the active Key Trustee Server: 
$ sudo -u keytrustee ktadmin enable-synchronous-replication --pg-rootdir /var/lib/keytrustee/db

If you modified the default database location, replace /var/lib/keytrustee/db with the modified path.

(Optional) Replace Self-Signed Certificates with CA-Signed Certificates

If you have a CA-signed certificate for Key Trustee Server, see Managing Key Trustee Server Certificates for instructions on how to replace the self-signed certificates.

Recovering a Key Trustee Server

If a Key Trustee Server fails, restore it from backup as soon as possible. If the Key Trustee Server hosts fails completely, make sure that you restore the Key Trustee Server to a new host with the same hostname and IP address as the failed host.

For more information, see Backing Up and Restoring Key Trustee Server.