Backing Up and Restoring Key Trustee Server

Key Trustee Server high availability applies to read operations only. If either Key Trustee Server fails, the client automatically retries fetching keys from the functioning server. New write operations (for example, creating new encryption keys) are not allowed unless both Key Trustee Servers are operational.

If a Key Trustee Server fails catastrophically, you must restore it from backup to a new host with the same hostname and IP address as the failed host. Cloudera does not support PostgreSQL promotion to convert a passive Key Trustee Server to an active Key Trustee Server.

Cloudera strongly recommends regularly backing up Key Trustee Server databases and configuration files. Because these backups contain encryption keys and encrypted deposits, you must ensure that your backup repository is as secure as the Key Trustee Server.

You must also back up client configuration files and keys for Key Trustee Server clients, such as Key Trusteee KMS and Navigator Encrypt clients.

Continue reading:

Backing Up Key Trustee Server

Use this procedure for both parcel-based and package-based installations.

If you have deployed Cloudera Navigator Key Trustee Server High Availability, perform these steps on both the active and passive Key Trustee Servers. The following procedure references the default database port and location; if you modified these settings during installation, replace the database and port with your values.
  1. Back up the Key Trustee Server database:
    • For Key Trustee Server 3.8: 
      $ su - postgres
$ pg_dump -c -p 5432 keytrustee | zip --encrypt keytrustee-db.zip -
    • For Key Trustee Server 5.4 and higher: 
      $ su - keytrustee
$ pg_dump -c -p 11381 keytrustee | zip --encrypt keytrustee-db.zip -

    The --encrypt option prompts you to create a password used to encrypt the zip file. This password is required to decrypt the file.

    For parcel-based installations, you must set environment variables after switching to the keytrustee user: 
    $ su - keytrustee
$ export PATH=$PATH:/opt/cloudera/parcels/KEYTRUSTEE_SERVER/PG_DB/opt/postgres/9.3/bin
$ export LD_LIBRARY_PATH=/opt/cloudera/parcels/KEYTRUSTEE_SERVER/PG_DB/opt/postgres/9.3/lib
$ pg_dump -c -p 11381 keytrustee | zip --encrypt keytrustee-db.zip -
  2. Back up the Key Trustee Server configuration directory (/var/lib/keytrustee/.keytrustee): 
    $ zip -r --encrypt keytrustee-conf.zip /var/lib/keytrustee/.keytrustee

    The --encrypt option prompts you to create a password used to encrypt the zip file. This password is required to decrypt the file.

  3. Move the backup files (keytrustee-db.zip and keytrustee-conf.zip) to a secure location.

Backing Up Key Trustee Server Clients

Cryptographic keys stored in Key Trustee Server are encrypted by clients before they are sent to Key Trustee Server. The primary clients for Key Trustee Server are Key Trustee KMS and Navigator Encrypt. Cloudera strongly recommends backing up regularly the configuration files and GPG keys for Key Trustee Server clients.

To prevent permanent data loss, regularly back up the following directories on each client that stores objects in Key Trustee Server:
Key Trustee Server Client Configuration Directories
Key Trustee Server Client Directories to Back Up
Key Trustee KMS /var/lib/kms-keytrustee
Navigator Encrypt /etc/navencrypt

Restoring Key Trustee Server

When restoring the Key Trustee Server database from backup, keep in mind that any keys or deposits created after the backup are not restored. If you are using Key Trustee Server high availability, you can restore the Active Key Trustee Server from the Passive Key Trustee Server. This restores all keys that were successfully written to the Passive Key Trustee Server before the failure.

The procedure to restore Key Trustee Server is different for parcel-based than for package-based installations. For more information about parcels, see Parcels.

Restoring Key Trustee Server in Parcel-Based Installations

If you have deployed Cloudera Navigator Key Trustee Server High Availability, perform these steps on both the active and passive Key Trustee Servers. The following procedures assume the default database port and location; if you modified these settings during installation, replace the database and port with your custom values.

If the Key Trustee Server host has failed completely, remove the host from the cluster and add a new host using Cloudera Manager:
  1. Remove the failed host from the cluster. See Deleting Hosts for instructions.
  2. Add a new host with the same hostname and IP address as the failed host to the cluster. See Adding a Host to the Cluster for instructions.
  3. Install Key Trustee Server on the new host. See Installing Cloudera Navigator Key Trustee Server for instructions. Make sure to install the same Key Trustee Server version as the failed host.

After you have provisioned a new host and installed Key Trustee Server (or if you are restoring the database or configuration on the original host), restore the database and configuration directory. If you need to restore the Active Key Trustee Server from the Passive Key Trustee Server, skip to Restoring Active Key Trustee Server from Passive Key Trustee Server.

If your backups were created manually using the pg_dump command, do the following:
  1. Copy or move the backup files (keytrustee-db.zip and keytrustee-conf.zip) to the Key Trustee Server host.
  2. Start the PostgreSQL server: 
    $ sudo ktadmin db --start --pg-rootdir /var/lib/keytrustee/db --background
  3. Restore the Key Trustee Server database: 
    $ su - keytrustee
$ export PATH=$PATH:/opt/cloudera/parcels/KEYTRUSTEE_SERVER/PG_DB/opt/postgres/9.3/bin
$ export LD_LIBRARY_PATH=/opt/cloudera/parcels/KEYTRUSTEE_SERVER/PG_DB/opt/postgres/9.3/lib
$ unzip -p /path/to/keytrustee-db.zip | psql -p 11381 -d keytrustee

    If the zip file is encrypted, you are prompted for the password to decrypt the file.

  4. Restore the Key Trustee Server configuration directory: 
    $ su - keytrustee
$ cd /var/lib/keytrustee
$ unzip /path/to/keytrustee-conf.zip

    If the zip file is encrypted, you are prompted for the password to decrypt the file.

  5. Stop the PostgreSQL server: 
    $ sudo ktadmin db --stop --pg-rootdir /var/lib/keytrustee/db
  6. Start the Key Trustee Server service in Cloudera Manager (Key Trustee Server service > Actions > Start).
  7. Remove the backup files (keytrustee-db.zip and keytrustee-conf.zip) from the Key Trustee Server host.

Restoring Key Trustee Server in Package-Based Installations

If you have deployed Cloudera Navigator Key Trustee Server High Availability, perform these steps on both the active and passive Key Trustee Servers. The following procedures assume the default database port and location; if you modified these settings during installation, replace the database and port with your custom values.

If the Key Trustee Server host has failed completely, provision a new host with the same hostname and IP address as the failed host, and re-install Key Trustee Server. See Installing Cloudera Navigator Key Trustee Server for instructions.

After you have provisioned a new host and installed Key Trustee Server (or if you are restoring the database or configuration on the original host), restore the database and configuration directory. If you need to restore the Active Key Trustee Server from the Passive Key Trustee Server, skip to Restoring Active Key Trustee Server from Passive Key Trustee Server.

If your backups were created manually using the pg_dump command, do the following:
  1. Copy or move the backup files (keytrustee-db.zip and keytrustee-conf.zip) to the Key Trustee Server host.
  2. Change the file ownership on the backup files to keytrustee:keytrustee: 
    $ sudo chown keytrustee:keytrustee /path/to/keytrustee*.zip
  3. Restore the Key Trustee Server database: 
    $ su - keytrustee
$ unzip -p /path/to/keytrustee-db.zip | psql -p 11381 -d keytrustee

    If the zip file is encrypted, you are prompted for the password to decrypt the file.

  4. Restore the Key Trustee Server configuration directory: 
    $ cd /var/lib/keytrustee
$ unzip /path/to/keytrustee-conf.zip

    If the zip file is encrypted, you are prompted for the password to decrypt the file.

  5. Start the Key Trustee Server service:
    • RHEL 6-compatible: $ sudo service keytrusteed start
    • RHEL 7-compatible: $ sudo systemctl start keytrusteed
  6. Remove the backup files (keytrustee-db.zip and keytrustee-conf.zip) from the Key Trustee Server host.

Restoring Active Key Trustee Server from Passive Key Trustee Server

If the Active Key Trustee Server fails, and you do not have a backup, you can restore it from the Passive Key Trustee Server using the following procedure. You can also use this procedure if you need to restore keys that were successfully written to the Passive Key Trustee Server, but are not included in the most recent backup.

The following procedure assumes you have installed Key Trustee Server on the replacement host and (if you are using Cloudera Manager) added the Key Trustee Server service. For instructions on installing Key Trustee Server, see Installing Cloudera Navigator Key Trustee Server

  1. Copy the Key Trustee Server database from the Passive Key Trustee Server host to the new Active Key Trustee Server host. Run the following command on the Passive Key Trustee Server host: 
    $ sudo rsync --exclude recovery.conf -a /var/lib/keytrustee/db root@kts01.example.com:/var/lib/keytrustee/

    Replace kts01.example.com with the hostname of the new Active Key Trustee Server.

  2. Make sure that the recovery.conf file did not get copied to the Active Key Trustee Server (for example, if there was a typo in your rsync command). Run the following command on the Active Key Trustee Server host: 
    $ sudo ls -l /var/lib/keytrustee/db/recovery.conf

    If the file exists on the Active Key Trustee Server host, delete it. Make sure you are on the Active Key Trustee Server host before deleting the file. Do not delete the recovery.conf file on the Passive Key Trustee Server host.

  3. Copy the configuration directory from the Passive Key Trustee Server host to the new Active Key Trustee Server host. Run the following command on the Passive Key Trustee Server host: 
    $ sudo rsync --exclude .ssl --exclude '*.pid' -a /var/lib/keytrustee/.keytrustee root@kts01.example.com:/var/lib/keytrustee/

    Replace kts01.example.com with the hostname of the new Active Key Trustee Server.

  4. Create the logs directory and make sure it is owned by the keytrustee user and group: 
    $ sudo mkdir /var/lib/keytrustee/logs
$ sudo chown keytrustee:keytrustee /var/lib/keytrustee/logs
  5. (Cloudera Manager only) Generate the Key Trustee Server keyring: Key Trustee Server service > Actions > Generate Key Trustee Server Keyring
  6. Set up the database on the Active Key Trustee Server host.
    • Using Cloudera Manager: Key Trustee Server service > Actions > Set Up Key Trustee Server Database
    • Using the Command Line: 
      $ sudo ktadmin --confdir /var/lib/keytrustee db --port 11381 --pg-rootdir /var/lib/keytrustee/db --bootstrap --slave kts02.example.com

      Replace kts02.example.com with the hostname of the Passive Key Trustee Server.

  7. Start the database.
    • Using Cloudera Manager: Key Trustee Server service > Instances > Active Database > Actions > Start this Active Database
    • Using the Command Line: Run the following command on the Active Key Trustee Server host: 
      $ sudo ktadmin --confdir /var/lib/keytrustee db --port 11381 --pg-rootdir /var/lib/keytrustee/db --bootstrap --slave kts02.example.com

      Replace kts02.example.com with the hostname of the Passive Key Trustee Server.

  8. Enable synchronous replication.
    • Using Cloudera Manager: Key Trustee Server service > Actions > Setup Enable Synchronous Replication in HA mode
    • Using the Command Line: Run the following command on the Active Key Trustee Server host: 
      $ sudo ktadmin --confdir /var/lib/keytrustee enable-synchronous-replication
  9. Restart the Active Key Trustee Server.
    • Using Cloudera Manager: Key Trustee Server service > Actions > Restart
    • Using the Command Line: Run the following command on the Active Key Trustee Server host: 
      $ sudo service keytrusteed restart      #RHEL 6-compatible
$ sudo systemctl restart keytrusteed    #RHEL 7-compatible