Backing Up and Restoring Key Trustee Server and Clients
Key Trustee Server high availability applies to read operations only. If either Key Trustee Server fails, the client automatically retries fetching keys from the functioning server. New write operations (for example, creating new encryption keys) are not allowed unless both Key Trustee Servers are operational.
If a Key Trustee Server fails catastrophically, you must restore it from backup to a new host with the same hostname and IP address as the failed host. Cloudera does not support PostgreSQL promotion to convert a passive Key Trustee Server to an active Key Trustee Server.
Cloudera strongly recommends regularly backing up Key Trustee Server databases and configuration files. Because these backups contain encryption keys and encrypted deposits, you must ensure that your backup repository is as secure as the Key Trustee Server.
You must also back up client configuration files and keys for Key Trustee Server clients, such as Key Trusteee KMS and Navigator Encrypt clients.
Backing Up Key Trustee Server and Key Trustee KMS Using the ktbackup.sh Script
Key Trustee Server 5.7 and higher includes a script, ktbackup.sh, to simplify and automate backing up Key Trustee Server. Key Trustee KMS 5.7 and higher include the same script for backing up Key Trustee KMS.
When run on a Key Trustee Server host, the script creates a tarball containing the Key Trustee Server private GPG keys and the PostgreSQL database. When run on a Key Trustee KMS host, the script creates a tarball containing the Key Trustee KMS private GPG keys and configuration file.
To preserve the security of the backup, you must specify a GPG recipient. Because this recipient is the only entity that can decrypt the backup, the recipient must be someone authorized to access the Key Trustee Server database, such as a key administrator.
Creating and Importing a GPG Key for Encrypting and Decrypting Backups
If the key administrator responsible for backing up and restoring Key Trustee Server and Key Trustee KMS does not already have a GPG key pair, they can create one using the gpg --gen-key command. The following example demonstrates this procedure:
[john.doe@backup-host ~]$ gpg --gen-key gpg (GnuPG) 2.0.14; Copyright (C) 2009 Free Software Foundation, Inc. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Please select what kind of key you want: (1) RSA and RSA (default) (2) DSA and Elgamal (3) DSA (sign only) (4) RSA (sign only) Your selection? 1 RSA keys may be between 1024 and 4096 bits long. What keysize do you want? (2048) Requested keysize is 2048 bits Please specify how long the key should be valid. 0 = key does not expire <n> = key expires in n days <n>w = key expires in n weeks <n>m = key expires in n months <n>y = key expires in n years Key is valid for? (0) Key does not expire at all Is this correct? (y/N) y GnuPG needs to construct a user ID to identify your key. Real name: John Doe Email address: john.doe@example.com Comment: Key Trustee Backup You selected this USER-ID: "John Doe (Key Trustee Backup) <john.doe@example.com>" Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O You need a Passphrase to protect your secret key. can't connect to `/home/john.doe/.gnupg/S.gpg-agent': No such file or directory gpg-agent[10638]: directory `/home/john.doe/.gnupg/private-keys-v1.d' created We need to generate a lot of random bytes. It is a good idea to perform some other action (type on the keyboard, move the mouse, utilize the disks) during the prime generation; this gives the random number generator a better chance to gain enough entropy. We need to generate a lot of random bytes. It is a good idea to perform some other action (type on the keyboard, move the mouse, utilize the disks) during the prime generation; this gives the random number generator a better chance to gain enough entropy. gpg: /home/john.doe/.gnupg/trustdb.gpg: trustdb created gpg: key 0936CB67 marked as ultimately trusted public and secret key created and signed. gpg: checking the trustdb gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u pub 2048R/0936CB67 2016-02-10 Key fingerprint = CE57 FDED 3AFE E67D 2041 9EBF E64B 7D00 0936 CB67 uid John Doe (Key Trustee Backup) <john.doe@example.com> sub 2048R/52A6FC5C 2016-02-10
After the GPG key pair is generated, you can export the public key:
[john.doe@backup-host ~]$ gpg --armor --output /path/to/johndoe.pub --export 'John Doe'
Copy the public key (johndoe.pub in this example) to the Key Trustee Server or Key Trustee KMS host, and import it into the service account keyring (keytrustee for Key Trustee Server and kms for Key Trustee KMS):
- On the Key Trustee Server host:
$ sudo -u keytrustee gpg --import /path/to/johndoe.pub
- On the Key Trustee KMS host:
$ sudo -u kms gpg --import /path/to/johndoe.pub
Running the ktbackup.sh Script
You must run ktbackup.sh as the service account. The location of the script depends on the service and installation method. See the following table for the script location and default service account for package- and parcel-based installations for Key Trustee Server and Key Trustee KMS.
Service | Service Account | Parcel-Based Installation | Package-Based Installation |
---|---|---|---|
Key Trustee Server | keytrustee | /opt/cloudera/parcels/KEYTRUSTEE_SERVER/bin/ktbackup.sh | /usr/bin/ktbackup.sh |
Key Trustee KMS | kms | /opt/cloudera/parcels/KEYTRUSTEE/bin/ktbackup.sh | /usr/share/keytrustee-keyprovider/bin/ktbackup.sh |
The following table lists the command options for ktbackup.sh.
Command Option | Description |
---|---|
-c, --confdir=CONFDIR | Specifies the Key Trustee configuration directory. Defaults to /var/lib/keytrustee/.keytrustee for parcel-based Key Trustee Server. For Key Trustee KMS and package-based Key Trustee Server, you must specify this option. |
--database-port=PORT | Specifies the Key Trustee Server database port. Defaults to 11381 for parcel-based installations. For package-based Key Trustee Server installations, you must specify this option. |
--gpg-recipient=GPG_RECIPIENT | Specifies the GPG recipient. The backup is encrypted with the public key of the specified recipient. The GPG recipient public key must be imported into the service account keyring before running the script. See Creating and Importing a GPG Key for Encrypting and Decrypting Backups for more information. |
--cleartext | Outputs an unencrypted tarball. To preserve the security of the cryptographic keys, do not use this option in production environments. |
--output=DIR | Specifies the output directory for the tarball. Defaults to /var/lib/keytrustee for parcel-based Key Trustee Server. For Key Trustee KMS and package-based Key Trustee Server, you must specify this option. |
-q, --quiet | Suppresses console log messages and, if successful, returns only the backup tarball file path. This is useful for automating backups. |
--verbose | Outputs additional log messages to the console for debugging. |
The following examples demonstrate the command usage for different scenarios:
- To back up a parcel-based Key Trustee Server, specifying the GPG recipient by name:
$ sudo -u keytrustee /opt/cloudera/parcels/KEYTRUSTEE_SERVER/bin/ktbackup.sh --gpg-recipient='John Doe'
- To back up a parcel-based Key Trustee KMS, specifying the GPG recipient by email:
$ sudo -u kms /opt/cloudera/parcels/KEYTRUSTEE/bin/ktbackup.sh -c /var/lib/kms-keytrustee/keytrustee/.keytrustee --output=/var/lib/kms-keytrustee --gpg-recipient=john.doe@example.com
- To back up a package-based Key Trustee Server with the database running on a nondefault port (12345 in this example):
$ sudo -u keytrustee ktbackup.sh --database-port=12345 --gpg-recipient=john.doe@example.com
- To back up a package-based Key Trustee KMS, specifying the GPG recipient by email:
$ sudo -u kms /usr/share/keytrustee-keyprovider/bin/ktbackup.sh -c /var/lib/kms-keytrustee/keytrustee/.keytrustee --output=/var/lib/kms-keytrustee --gpg-recipient=john.doe@example.com
Automating Backups Using cron
You can schedule automatic backups of Key Trustee Server or Key Trustee KMS using the cron scheduling utility.
Create a crontab entry using the following commands:
- For Key Trustee Server:
- Edit the crontab by running the following command:
$ sudo -u keytrustee crontab -e
- Add the following entry to run the backup script every 30 minutes. This example is for a parcel-based installation of Key Trustee Server. See the Backup Script Locations table for the package-based script location.
*/30 * * * * /opt/cloudera/parcels/KEYTRUSTEE_SERVER/bin/ktbackup.sh --gpg-recipient='John Doe' --quiet --output=/tmp/backups
Run man 5 crontab to see the crontab man page for details on using cron to schedule backups at different intervals.
- Edit the crontab by running the following command:
- For Key Trustee KMS:
- Edit the crontab by running the following command:
$ sudo -u kms crontab -e
- Add the following entry to run the backup script every 30 minutes. This example is for a parcel-based installation of Key Trustee KMS. See the Backup Script Locations table for the package-based script location.
*/30 * * * * /opt/cloudera/parcels/KEYTRUSTEE/bin/ktbackup.sh --gpg-recipient='John Doe' --quiet --output=/tmp/backups
Run man 5 crontab to see the crontab man page for details on using cron to schedule backups at different intervals.
- Edit the crontab by running the following command:
Backing Up Key Trustee Server Manually
Use this procedure for both parcel-based and package-based installations.
- Back up the Key Trustee Server database:
- For Key Trustee Server 3.8:
$ su - postgres $ pg_dump -c -p 5432 keytrustee | zip --encrypt keytrustee-db.zip -
- For Key Trustee Server 5.4 and higher:
$ su - keytrustee $ pg_dump -c -p 11381 keytrustee | zip --encrypt keytrustee-db.zip -
The --encrypt option prompts you to create a password used to encrypt the zip file. This password is required to decrypt the file.
For parcel-based installations, you must set environment variables after switching to the keytrustee user:$ su - keytrustee $ export PATH=$PATH:/opt/cloudera/parcels/KEYTRUSTEE_SERVER/PG_DB/opt/postgres/9.3/bin $ export LD_LIBRARY_PATH=/opt/cloudera/parcels/KEYTRUSTEE_SERVER/PG_DB/opt/postgres/9.3/lib $ pg_dump -c -p 11381 keytrustee | zip --encrypt keytrustee-db.zip -
- For Key Trustee Server 3.8:
- Back up the Key Trustee Server configuration directory (/var/lib/keytrustee/.keytrustee):
$ zip -r --encrypt keytrustee-conf.zip /var/lib/keytrustee/.keytrustee
The --encrypt option prompts you to create a password used to encrypt the zip file. This password is required to decrypt the file.
- Move the backup files (keytrustee-db.zip and keytrustee-conf.zip) to a secure location.
Backing Up Key Trustee Server Clients
Cryptographic keys stored in Key Trustee Server are encrypted by clients before they are sent to Key Trustee Server. The primary clients for Key Trustee Server are Key Trustee KMS and Navigator Encrypt. Cloudera strongly recommends backing up regularly the configuration files and GPG keys for Key Trustee Server clients. See Backing Up Key Trustee Server and Key Trustee KMS Using the ktbackup.sh Script for instructions on backing up Key Trustee KMS using the provided backup script.
Key Trustee Server Client | Directories to Back Up |
---|---|
Key Trustee KMS | /var/lib/kms-keytrustee |
Navigator Encrypt | /etc/navencrypt |
Restoring Key Trustee Server
When restoring the Key Trustee Server database from backup, keep in mind that any keys or deposits created after the backup are not restored. If you are using Key Trustee Server high availability, you can restore the Active Key Trustee Server from the Passive Key Trustee Server. This restores all keys that were successfully written to the Passive Key Trustee Server before the failure.
The procedure to restore Key Trustee Server is different for parcel-based than for package-based installations. For more information about parcels, see Parcels.
Restoring Key Trustee Server in Parcel-Based Installations
If you have deployed Cloudera Navigator Key Trustee Server High Availability, perform these steps on both the active and passive Key Trustee Servers. The following procedures assume the default database port and location; if you modified these settings during installation, replace the database and port with your custom values.
- Remove the failed host from the cluster. See Deleting Hosts for instructions.
- Add a new host with the same hostname and IP address as the failed host to the cluster. See Adding a Host to the Cluster for instructions.
- Install Key Trustee Server on the new host. See Installing Cloudera Navigator Key Trustee Server for instructions. Make sure to install the same Key Trustee Server version as the failed host.
After you have provisioned a new host and installed Key Trustee Server (or if you are restoring the database or configuration on the original host), restore the database and configuration directory. If your backups were created using the ktbackup.sh script, skip to Restoring Key Trustee Server and Key Trustee KMS from ktbackup.sh Backups. If you need to restore the Active Key Trustee Server from the Passive Key Trustee Server, skip to Restoring Active Key Trustee Server from Passive Key Trustee Server.
- Copy or move the backup files (keytrustee-db.zip and keytrustee-conf.zip) to the Key Trustee Server host.
- Start the PostgreSQL server:
$ sudo ktadmin db --start --pg-rootdir /var/lib/keytrustee/db --background
- Restore the Key Trustee Server database:
$ su - keytrustee $ export PATH=$PATH:/opt/cloudera/parcels/KEYTRUSTEE_SERVER/PG_DB/opt/postgres/9.3/bin $ export LD_LIBRARY_PATH=/opt/cloudera/parcels/KEYTRUSTEE_SERVER/PG_DB/opt/postgres/9.3/lib $ unzip -p /path/to/keytrustee-db.zip | psql -p 11381 -d keytrustee
If the zip file is encrypted, you are prompted for the password to decrypt the file.
- Restore the Key Trustee Server configuration directory:
$ su - keytrustee $ cd /var/lib/keytrustee $ unzip /path/to/keytrustee-conf.zip
If the zip file is encrypted, you are prompted for the password to decrypt the file.
- Stop the PostgreSQL server:
$ sudo ktadmin db --stop --pg-rootdir /var/lib/keytrustee/db
- Start the Key Trustee Server service in Cloudera Manager ( ).
- Restart the Key Trustee KMS service in Cloudera Manager ( ).
- Remove the backup files (keytrustee-db.zip and keytrustee-conf.zip) from the Key Trustee Server host.
Restoring Key Trustee Server in Package-Based Installations
If you have deployed Cloudera Navigator Key Trustee Server High Availability, perform these steps on both the active and passive Key Trustee Servers. The following procedures assume the default database port and location; if you modified these settings during installation, replace the database and port with your custom values.
After you have provisioned a new host and installed Key Trustee Server (or if you are restoring the database or configuration on the original host), restore the database and configuration directory. If your backups were created using the ktbackup.sh script, skip to Restoring Key Trustee Server and Key Trustee KMS from ktbackup.sh Backups. If you need to restore the Active Key Trustee Server from the Passive Key Trustee Server, skip to Restoring Active Key Trustee Server from Passive Key Trustee Server.
- Copy or move the backup files (keytrustee-db.zip and keytrustee-conf.zip) to the Key Trustee Server host.
- Change the file ownership on the backup files to keytrustee:keytrustee:
$ sudo chown keytrustee:keytrustee /path/to/keytrustee*.zip
- Restore the Key Trustee Server database:
$ su - keytrustee $ unzip -p /path/to/keytrustee-db.zip | psql -p 11381 -d keytrustee
If the zip file is encrypted, you are prompted for the password to decrypt the file.
- Restore the Key Trustee Server configuration directory:
$ cd /var/lib/keytrustee $ unzip /path/to/keytrustee-conf.zip
If the zip file is encrypted, you are prompted for the password to decrypt the file.
- Start the Key Trustee Server service:
- RHEL 6-compatible: $ sudo service keytrusteed start
- RHEL 7-compatible: $ sudo systemctl start keytrusteed
- Remove the backup files (keytrustee-db.zip and keytrustee-conf.zip) from the Key Trustee Server host.
Restoring Key Trustee Server and Key Trustee KMS from ktbackup.sh Backups
- Decrypt the backup tarball using the private key of the GPG recipient specified in the backup command by running the following command as the GPG recipient user account. The GPG
recipient private key must be available on the Key Trustee Server or Key Trustee KMS host on which you run this command.
$ gpg -d -o /path/to/decrypted/backup.tar /path/to/encrypted/tarball
- Verify the decrypted tarball using the tar tvf /path/to/decrypted/backup.tar command. For example:
$ tar tvf kts_bak_kts01_example_com_2016-02-10_11-14-37.tar drwx------ keytrustee/keytrustee 0 2016-02-09 16:43 var/lib/keytrustee/.keytrustee/ -rw------- keytrustee/keytrustee 434 2016-02-09 16:43 var/lib/keytrustee/.keytrustee/keytrustee.conf -rw------- keytrustee/keytrustee 1280 2016-02-09 16:43 var/lib/keytrustee/.keytrustee/trustdb.gpg -rw------- keytrustee/keytrustee 4845 2016-02-09 16:43 var/lib/keytrustee/.keytrustee/secring.gpg -rw------- keytrustee/keytrustee 600 2016-02-09 16:43 var/lib/keytrustee/.keytrustee/random_seed drwx------ keytrustee/keytrustee 0 2016-02-09 16:40 var/lib/keytrustee/.keytrustee/.ssl/ -rw------- keytrustee/keytrustee 1708 2016-02-09 16:40 var/lib/keytrustee/.keytrustee/.ssl/ssl-cert-keytrustee-pk.pem -rw------- keytrustee/keytrustee 1277 2016-02-09 16:40 var/lib/keytrustee/.keytrustee/.ssl/ssl-cert-keytrustee.pem -rw------- keytrustee/keytrustee 2263 2016-02-09 16:43 var/lib/keytrustee/.keytrustee/pubring.gpg -rw-r--r-- keytrustee/keytrustee 457 2016-02-09 16:43 var/lib/keytrustee/.keytrustee/logging.conf -rw------- keytrustee/keytrustee 2263 2016-02-09 16:43 var/lib/keytrustee/.keytrustee/pubring.gpg~ -rw------- keytrustee/keytrustee 157 2016-02-09 16:40 var/lib/keytrustee/.keytrustee/gpg.conf -rw-r--r-- keytrustee/keytrustee 47752 2016-02-10 11:14 var/lib/keytrustee/kts_bak_kts01_example_com_2016-02-10_11-14-37.sql
- Restore the files to their original locations, using this command for both Key Trustee Server and Key Trustee KMS backups:
$ tar xvf /path/to/decrypted/backup.tar -C /
- (Key Trustee Server Only) Drop and re-create the keytrustee PostgreSQL database, and restore the database from the backup.
- For parcel-based installations:
$ su - keytrustee $ source /opt/cloudera/parcels/KEYTRUSTEE_SERVER/meta/keytrustee_env.sh $ /opt/cloudera/parcels/KEYTRUSTEE_SERVER/PG_DB/opt/postgres/9.3/bin/psql -p 11381 psql (9.3.6) Type "help" for help. keytrustee=# \list List of databases Name | Owner | Encoding | Collate | Ctype | Access privileges ------------+------------+----------+-------------+-------------+--------------------------- keytrustee | keytrustee | UTF8 | en_US.UTF-8 | en_US.UTF-8 | postgres | keytrustee | UTF8 | en_US.UTF-8 | en_US.UTF-8 | template0 | keytrustee | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/keytrustee + | | | | | keytrustee=CTc/keytrustee template1 | keytrustee | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/keytrustee + | | | | | keytrustee=CTc/keytrustee (4 rows) keytrustee=# \c postgres; You are now connected to database "postgres" as user "keytrustee". postgres=# drop database keytrustee; DROP DATABASE postgres=# create database keytrustee; CREATE DATABASE postgres=# \q $ sudo -u keytrustee /opt/cloudera/parcels/KEYTRUSTEE_SERVER/PG_DB/opt/postgres/9.3/bin/psql -p 11381 -f /var/lib/keytrustee/kts_bak_kts01_example_com_2016-02-10_11-14-37.sql
- For package-based installations:
$ su - keytrustee $ psql -p 11381 psql (9.3.6) Type "help" for help. keytrustee=# \list List of databases Name | Owner | Encoding | Collate | Ctype | Access privileges ------------+------------+----------+-------------+-------------+--------------------------- keytrustee | keytrustee | UTF8 | en_US.UTF-8 | en_US.UTF-8 | postgres | keytrustee | UTF8 | en_US.UTF-8 | en_US.UTF-8 | template0 | keytrustee | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/keytrustee + | | | | | keytrustee=CTc/keytrustee template1 | keytrustee | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/keytrustee + | | | | | keytrustee=CTc/keytrustee (4 rows) keytrustee=# \c postgres; You are now connected to database "postgres" as user "keytrustee". postgres=# drop database keytrustee; DROP DATABASE postgres=# create database keytrustee; CREATE DATABASE postgres=# \q $ sudo -u keytrustee psql -p 11381 -f /var/lib/keytrustee/kts_bak_kts01_example_com_2016-02-10_11-14-37.sql
- For parcel-based installations:
- Restart Key Trustee Server.
- Using Cloudera Manager:
- Using the Command Line: Run the following command on the Key Trustee Server hosts:
$ sudo service keytrusteed restart #RHEL 6-compatible $ sudo systemctl restart keytrusteed #RHEL 7-compatible
- Restart the Key Trustee KMS service in Cloudera Manager ( ).
Restoring Active Key Trustee Server from Passive Key Trustee Server
If the Active Key Trustee Server fails, and you do not have a backup, you can restore it from the Passive Key Trustee Server using the following procedure. You can also use this procedure if you need to restore keys that were successfully written to the Passive Key Trustee Server, but are not included in the most recent backup.
The following procedure assumes you have installed Key Trustee Server on the replacement host and (if you are using Cloudera Manager) added the Key Trustee Server service. For instructions on installing Key Trustee Server, see Installing Cloudera Navigator Key Trustee Server
- Copy the Key Trustee Server database from the Passive Key Trustee Server host to the new Active Key Trustee Server host. Run the following command on the Passive Key Trustee Server
host:
$ sudo rsync --exclude recovery.conf -a /var/lib/keytrustee/db root@kts01.example.com:/var/lib/keytrustee/
Replace kts01.example.com with the hostname of the new Active Key Trustee Server.
- Make sure that the recovery.conf file did not get copied to the Active Key Trustee Server (for example, if there was a typo in your rsync command). Run the following command on the Active Key Trustee Server host:
$ sudo ls -l /var/lib/keytrustee/db/recovery.conf
If the file exists on the Active Key Trustee Server host, delete it. Make sure you are on the Active Key Trustee Server host before deleting the file. Do not delete the recovery.conf file on the Passive Key Trustee Server host.
- Copy the configuration directory from the Passive Key Trustee Server host to the new Active Key Trustee Server host. Run the following command on the Passive Key Trustee Server host:
$ sudo rsync --exclude .ssl --exclude '*.pid' -a /var/lib/keytrustee/.keytrustee root@kts01.example.com:/var/lib/keytrustee/
Replace kts01.example.com with the hostname of the new Active Key Trustee Server.
- Create the logs directory and make sure it is owned by the keytrustee user and group:
$ sudo mkdir /var/lib/keytrustee/logs $ sudo chown keytrustee:keytrustee /var/lib/keytrustee/logs
- (Cloudera Manager only) Generate the Key Trustee Server keyring:
- Set up the database on the Active Key Trustee Server host.
- Using Cloudera Manager:
- Using the Command Line:
$ sudo ktadmin --confdir /var/lib/keytrustee db --port 11381 --pg-rootdir /var/lib/keytrustee/db --bootstrap --slave kts02.example.com
Replace kts02.example.com with the hostname of the Passive Key Trustee Server.
- Start the database.
- Using Cloudera Manager:
- Using the Command Line: Run the following command on the Active Key Trustee Server host:
$ sudo ktadmin --confdir /var/lib/keytrustee db --port 11381 --pg-rootdir /var/lib/keytrustee/db --bootstrap --slave kts02.example.com
Replace kts02.example.com with the hostname of the Passive Key Trustee Server.
- Enable synchronous replication.
- Using Cloudera Manager:
- Using the Command Line: Run the following command on the Active Key Trustee Server host:
$ sudo ktadmin --confdir /var/lib/keytrustee enable-synchronous-replication
- Restart the Active Key Trustee Server.
- Using Cloudera Manager:
- Using the Command Line: Run the following command on the Active Key Trustee Server host:
$ sudo service keytrusteed restart #RHEL 6-compatible $ sudo systemctl restart keytrusteed #RHEL 7-compatible
- Restart the Key Trustee KMS service in Cloudera Manager ( ).