Backing up Key Trustee Server and clients

In case of failure, you should regularly back up Key Trustee Server databases and configuration files. You must also back up client configuration files and keys for Key Trustee Server clients, such as Key Trustee KMS and Navigator Encrypt clients.

Key Trustee Server high availability applies to read operations only. If either Key Trustee Server fails, the client automatically retries fetching keys from the functioning server. New write operations (for example, creating new encryption keys) are not allowed unless both Key Trustee Servers are operational.

Cloudera strongly recommends regularly backing up Key Trustee Server databases and configuration files. Because these backups contain encryption keys and encrypted deposits, you must ensure that your backup repository is as secure as the Key Trustee Server.

You must also back up client configuration files and keys for Key Trustee Server clients, such as Key Trustee KMS and Navigator Encrypt clients.

Back up Key Trustee Server and Key Trustee KMS using Cloudera Manager

Cloudera Manager versions 5.8 and higher, when used with Key Trustee Server and Key Trustee KMS versions 5.7 and higher, allow for backups of the KT Server and KT KMS configurations.

The actions executed in this procedure are equivalent to running the ktbackup.sh script on the node in question (see Back up Key Trustee Server and Key Trustee KMS using the ktbackup.sh script for additional details.

In addition, when using the HDFS Encryption Wizard in Cloudera Manager 5.8 or higher to install and configure Key Trustee Server and Key Trustee KMS versions 5.7 and higher, a cron job is automatically set up to back up the Key Trustee Server on an ongoing basis. See Initializing Standalone Key Trustee Server for more detail.

To back up the KT Server or KT KMS service configuration using Cloudera Manager:

Select the KT Server or KMS service configuration that you wish to back up.
For a KT Server backup, select Create Backup on Active Server (or Create Backup on Passive Server) from the Actions menu. For a KMS backup, select Create Backup.

A successfully completed backup of the KT Server is indicated by the message “Command Create Backup on Active Server finished successfully on service keytrustee_server”.

Back up Key Trustee Server and Key Trustee KMS using the ktbackup.sh script

Key Trustee Server releases 5.7 and higher include a script, ktbackup.sh, to simplify and automate backing up Key Trustee Server. Key Trustee KMS releases 5.7 and higher include the same script for backing up Key Trustee KMS.

When run on a Key Trustee Server host, the script creates a tarball containing the Key Trustee Server private GPG keys and the PostgreSQL database. When run on a Key Trustee KMS host, the script creates a tarball containing the Key Trustee KMS private GPG keys and configuration file.

To preserve the security of the backup, you must specify a GPG recipient. Because this recipient is the only entity that can decrypt the backup, the recipient must be someone authorized to access the Key Trustee Server database, such as a key administrator.

Creating and Importing a GPG Key for Encrypting and Decrypting Backups

If the key administrator responsible for backing up and restoring Key Trustee Server and Key Trustee KMS does not already have a GPG key pair, they can create one using the gpg --gen-key command. The following example demonstrates this procedure:

[john.doe@backup-host ~]$ gpg --gen-key
gpg (GnuPG) 2.0.14; Copyright (C) 2009 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
Your selection? 1
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048)
Requested keysize is 2048 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0)
Key does not expire at all
Is this correct? (y/N) y

GnuPG needs to construct a user ID to identify your key.

Real name: John Doe
Email address: john.doe@example.com
Comment: Key Trustee Backup
You selected this USER-ID:
    "John Doe (Key Trustee Backup) <john.doe@example.com>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
You need a Passphrase to protect your secret key.

can't connect to `/home/john.doe/.gnupg/S.gpg-agent': No such file or directory
gpg-agent[10638]: directory `/home/john.doe/.gnupg/private-keys-v1.d' created
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: /home/john.doe/.gnupg/trustdb.gpg: trustdb created
gpg: key 0936CB67 marked as ultimately trusted
public and secret key created and signed.

gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
pub   2048R/0936CB67 2016-02-10
      Key fingerprint = CE57 FDED 3AFE E67D 2041  9EBF E64B 7D00 0936 CB67
uid                  John Doe (Key Trustee Backup) <john.doe@example.com>
sub   2048R/52A6FC5C 2016-02-10

After the GPG key pair is generated, you can export the public key:

[john.doe@backup-host ~]$ gpg --armor --output /path/to/johndoe.pub --export 'John Doe'

Copy the public key (johndoe.pub in this example) to the Key Trustee Server or Key Trustee KMS host, and import it into the service account keyring (keytrustee for Key Trustee Server and kms for Key Trustee KMS):

On the Key Trustee Server host:

sudo -u keytrustee gpg --import /path/to/johndoe.pub

On the Key Trustee KMS host:

sudo -u kms gpg --import /path/to/johndoe.pub

Running the ktbackup.sh Script

You must run ktbackup.sh as the service account. The location of the script depends on the service and installation method. See the following table for the script location and default service account for package- and parcel-based installations for Key Trustee Server and Key Trustee KMS.

Table 1. Backup Script Locations
Service	Service Account	Parcel-Based Installation	Package-Based Installation
Key Trustee Server	`keytrustee`	`/opt/cloudera/parcels/KEYTRUSTEE_SERVER/bin/ktbackup.sh`	`/usr/bin/ktbackup.sh`
Key Trustee KMS	`kms`	`/opt/cloudera/parcels/KEYTRUSTEE/bin/ktbackup.sh`	`/usr/share/keytrustee-keyprovider/bin/ktbackup.sh`

The following table lists the command options for ktbackup.sh.

Table 2. Command Options for ktbackup.sh
Command Option	Description
`-c, --confdir=CONFDIR`	Specifies the Key Trustee configuration directory. Defaults to `/var/lib/keytrustee/.keytrustee` for parcel-based Key Trustee Server. For Key Trustee KMS and package-based Key Trustee Server, you must specify this option.
`--database-port=PORT`	Specifies the Key Trustee Server database port. Defaults to `11381` for parcel-based installations. For package-based Key Trustee Server installations, you must specify this option.
`--gpg-recipient=GPG_RECIPIENT`	Specifies the GPG recipient. The backup is encrypted with the public key of the specified recipient. The GPG recipient public key must be imported into the service account keyring before running the script. See Creating and Importing a GPG Key for Encrypting and Decrypting Backups for more information.
`--cleartext`	Outputs an unencrypted tarball. To preserve the security of the cryptographic keys, do not use this option in production environments.
`--output=DIR`	Specifies the output directory for the tarball. Defaults to `/var/lib/keytrustee` for parcel-based Key Trustee Server. For Key Trustee KMS and package-based Key Trustee Server, you must specify this option.
`--roll=n`	Deletes backups older than the last `n` backups from the directory specified by the `--output` parameter. For example, if you have 10 backups, specifying `--roll=10` creates a new backup (11 backups total) and then delete the oldest backup. Specifying `--roll=1` creates a new backup and then deletes all other backups. note This option works for Key Trustee Server only.
`-q, --quiet`	Suppresses console log messages and, if successful, returns only the backup tarball file path. This is useful for automating backups.
`--verbose`	Outputs additional log messages to the console for debugging.

The following examples demonstrate the command usage for different scenarios:

To back up a parcel-based Key Trustee Server, specifying the GPG recipient by name:
```
$ sudo -u keytrustee /opt/cloudera/parcels/KEYTRUSTEE_SERVER/bin/ktbackup.sh --gpg-recipient='John Doe'
```

To back up a parcel-based Key Trustee KMS, specifying the GPG recipient by email:

$ sudo -u kms /opt/cloudera/parcels/KEYTRUSTEE/bin/ktbackup.sh -c /var/lib/kms-keytrustee/keytrustee/.keytrustee --output=/var/lib/kms-keytrustee --gpg-recipient=john.doe@example.com

To back up a package-based Key Trustee Server with the database running on a non-default port (12345 in this example):
```
$ sudo -u keytrustee ktbackup.sh --database-port=12345 --gpg-recipient=john.doe@example.com
```

To back up a package-based Key Trustee KMS, specifying the GPG recipient by email:

$ sudo -u kms /usr/share/keytrustee-keyprovider/bin/ktbackup.sh -c /var/lib/kms-keytrustee/keytrustee/.keytrustee --output=/var/lib/kms-keytrustee --gpg-recipient=john.doe@example.com

Automating Backups Using cron

You can schedule automatic backups of Key Trustee Server using the cron scheduling utility.

Create a crontab entry using the following commands:

Edit the crontab by running the following command:
```
sudo -u keytrustee crontab -e
```
Add the following entry to run the backup script every 30 minutes. This example is for a parcel-based installation of Key Trustee Server. See the Backup Script Locations table for the package-based script location.
```
*/30 * * * * /opt/cloudera/parcels/KEYTRUSTEE_SERVER/bin/ktbackup.sh --gpg-recipient='John Doe' --quiet --output=/tmp/backups --roll=10
```
Run man 5 crontab to see the crontab man page for details on using cron to schedule backups at different intervals.

Back up Key Trustee Server manually

Use this procedure for both parcel-based and package-based installations.

The following procedure references the default database port and location; if you modified these settings during installation, replace the database and port with your values.

Back up the Key Trustee Server database:

For Key Trustee Server 3.8:

su - postgres
pg_dump -c -p 5432 keytrustee | zip --encrypt keytrustee-db.zip -

For Key Trustee Server 5.4 and higher:

su - keytrustee
pg_dump -c -p 11381 keytrustee | zip --encrypt keytrustee-db.zip -

The --encrypt option prompts you to create a password used to encrypt the zip file. This password is required to decrypt the file.

For parcel-based installations, you must set environment variables after switching to the keytrustee user:

su - keytrustee
export PATH=$PATH:/opt/cloudera/parcels/KEYTRUSTEE_SERVER/PG_DB/opt/postgres/9.3/bin
export LD_LIBRARY_PATH=/opt/cloudera/parcels/KEYTRUSTEE_SERVER/PG_DB/opt/postgres/9.3/lib
pg_dump -c -p 11381 keytrustee | zip --encrypt keytrustee-db.zip -

Back up the Key Trustee Server configuration directory (/var/lib/keytrustee/.keytrustee):
```
zip -r --encrypt keytrustee-conf.zip /var/lib/keytrustee/.keytrustee
```
The --encrypt option prompts you to create a password used to encrypt the zip file. This password is required to decrypt the file.
Move the backup files (keytrustee-db.zip and keytrustee-conf.zip) to a secure location.

Back up Key Trustee Server clients

Cryptographic keys stored in Key Trustee Server are encrypted by clients before they are sent to Key Trustee Server. The primary clients for Key Trustee Server are Key Trustee KMS and Navigator Encrypt. Cloudera strongly recommends backing up regularly the configuration files and GPG keys for Key Trustee Server clients. See Back up Key Trustee Server and Key Trustee KMS using the ktbackup.sh script for instructions on backing up Key Trustee KMS using the provided backup script.

To prevent permanent data loss, regularly back up the following directories on each client that stores objects in Key Trustee Server:

Table 3. Key Trustee Server Client Configuration Directories
Key Trustee Server Client	Directories to Back Up
Key Trustee KMS	`/var/lib/kms-keytrustee`
Navigator Encrypt	`/etc/navencrypt`

Restoring Key Trustee Server

Restore Key Trustee Server in parcel-based installations

The following procedures assume the default database port and location; if you modified these settings during installation, replace the database and port with your custom values.

If the Key Trustee Server host has failed completely, remove the host from the cluster and add a new host using Cloudera Manager:

Remove the failed host from the cluster.
Add a new host with the same hostname and IP address as the failed host to the cluster.
important
Make sure that the replacement host uses the same operating system version as the failed host.
Install Key Trustee Server on the new host. Make sure to install the same Key Trustee Server version as the failed host.

After you have provisioned a new host and installed Key Trustee Server (or if you are restoring the database or configuration on the original host), restore the database and configuration directory. If your backups were created using the ktbackup.sh script, skip to Restore Key Trustee Server and Key Trustee KMS from ktbackup.sh backups. If you need to restore the Active Key Trustee Server from the Passive Key Trustee Server, skip to Restore Active Key Trustee Server from Passive Key Trustee Server.

If your backups were created manually using the pg_dump command, do the following:

Copy or move the backup files (keytrustee-db.zip and keytrustee-conf.zip) to the Key Trustee Server host.

Start the PostgreSQL server:

sudo ktadmin db --start --pg-rootdir /var/lib/keytrustee/db --background

Restore the Key Trustee Server database:

su - keytrustee
export PATH=$PATH:/opt/cloudera/parcels/KEYTRUSTEE_SERVER/PG_DB/opt/postgres/9.3/bin
export LD_LIBRARY_PATH=/opt/cloudera/parcels/KEYTRUSTEE_SERVER/PG_DB/opt/postgres/9.3/lib
unzip -p /path/to/keytrustee-db.zip | psql -p 11381 -d keytrustee

If the zip file is encrypted, you are prompted for the password to decrypt the file.

Restore the Key Trustee Server configuration directory:
```
su - keytrustee
cd /var/lib/keytrustee
unzip /path/to/keytrustee-conf.zip
```
If the zip file is encrypted, you are prompted for the password to decrypt the file.

Stop the PostgreSQL server:

sudo ktadmin db --stop --pg-rootdir /var/lib/keytrustee/db

Start the Key Trustee Server service in Cloudera Manager (Key Trustee Server service > Actions > Start).
Restart the Key Trustee KMS service in Cloudera Manager (Key Trustee KMS service > Actions > Restart).
Remove the backup files (keytrustee-db.zip and keytrustee-conf.zip) from the Key Trustee Server host.

Restore Key Trustee Server in package-based installations

The following procedures assume the default database port and location; if you modified these settings during installation, replace the database and port with your custom values.

If the Key Trustee Server host has failed completely, provision a new host with the same hostname and IP address as the failed host, and re-install Key Trustee Server.

If your backups were created manually using the pg_dump command, do the following:

Copy or move the backup files (keytrustee-db.zip and keytrustee-conf.zip) to the Key Trustee Server host.
Change the file ownership on the backup files to keytrustee:keytrustee:
```
sudo chown keytrustee:keytrustee /path/to/keytrustee*.zip
```
Restore the Key Trustee Server database:
```
su - keytrustee
unzip -p /path/to/keytrustee-db.zip | psql -p 11381 -d keytrustee
```
If the zip file is encrypted, you are prompted for the password to decrypt the file.
Restore the Key Trustee Server configuration directory:
```
cd /var/lib/keytrustee
unzip /path/to/keytrustee-conf.zip
```
If the zip file is encrypted, you are prompted for the password to decrypt the file.
Start the Key Trustee Server service:
- RHEL 6-compatible: $ sudo service keytrusteed start
- RHEL 7-compatible: $ sudo systemctl start keytrusteed
Remove the backup files (keytrustee-db.zip and keytrustee-conf.zip) from the Key Trustee Server host.

Restore Key Trustee Server and Key Trustee KMS from ktbackup.sh backups

After installing Key Trustee Server or Key Trustee KMS on a new host after a failure, or if you need to restore accidentally deleted keys on the same host, use the following procedure to restore Key Trustee Server or Key Trustee KMS from backups generated by the ktbackup.sh script.

Decrypt the backup tarball using the private key of the GPG recipient specified in the backup command by running the following command as the GPG recipient user account. The GPG recipient private key must be available on the Key Trustee Server or Key Trustee KMS host on which you run this command.
```
gpg -d -o /path/to/decrypted/backup.tar /path/to/encrypted/tarball
```

Verify the decrypted tarball using the

tar tvf
                  /path/to/decrypted/backup.tar

command. For example:

$ tar tvf kts_bak_kts01_example_com_2016-02-10_11-14-37.tar
drwx------ keytrustee/keytrustee 0 2016-02-09 16:43 var/lib/keytrustee/.keytrustee/
-rw------- keytrustee/keytrustee 434 2016-02-09 16:43 var/lib/keytrustee/.keytrustee/keytrustee.conf
-rw------- keytrustee/keytrustee 1280 2016-02-09 16:43 var/lib/keytrustee/.keytrustee/trustdb.gpg
-rw------- keytrustee/keytrustee 4845 2016-02-09 16:43 var/lib/keytrustee/.keytrustee/secring.gpg
-rw------- keytrustee/keytrustee  600 2016-02-09 16:43 var/lib/keytrustee/.keytrustee/random_seed
drwx------ keytrustee/keytrustee    0 2016-02-09 16:40 var/lib/keytrustee/.keytrustee/.ssl/
-rw------- keytrustee/keytrustee 1708 2016-02-09 16:40 var/lib/keytrustee/.keytrustee/.ssl/ssl-cert-keytrustee-pk.pem
-rw------- keytrustee/keytrustee 1277 2016-02-09 16:40 var/lib/keytrustee/.keytrustee/.ssl/ssl-cert-keytrustee.pem
-rw------- keytrustee/keytrustee 2263 2016-02-09 16:43 var/lib/keytrustee/.keytrustee/pubring.gpg
-rw-r--r-- keytrustee/keytrustee  457 2016-02-09 16:43 var/lib/keytrustee/.keytrustee/logging.conf
-rw------- keytrustee/keytrustee 2263 2016-02-09 16:43 var/lib/keytrustee/.keytrustee/pubring.gpg~
-rw------- keytrustee/keytrustee  157 2016-02-09 16:40 var/lib/keytrustee/.keytrustee/gpg.conf
-rw-r--r-- keytrustee/keytrustee 47752 2016-02-10 11:14 var/lib/keytrustee/kts_bak_kts01_example_com_2016-02-10_11-14-37.sql

Restore the files to their original locations, using this command for both Key Trustee Server and Key Trustee KMS backups:
```
tar xvf /path/to/decrypted/backup.tar -C /
```

(Key Trustee Server Only) Drop and re-create the keytrustee PostgreSQL database, and restore the database from the backup.

For parcel-based installations:

$ su - keytrustee
$ source /opt/cloudera/parcels/KEYTRUSTEE_SERVER/meta/keytrustee_env.sh
$ /opt/cloudera/parcels/KEYTRUSTEE_SERVER/PG_DB/opt/postgres/9.3/bin/psql -p 11381
psql (9.3.6)
Type "help" for help.

keytrustee=# \list
                                     List of databases
    Name    |   Owner    | Encoding |   Collate   |    Ctype    |     Access privileges
------------+------------+----------+-------------+-------------+---------------------------
 keytrustee | keytrustee | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 postgres   | keytrustee | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 template0  | keytrustee | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/keytrustee            +
            |            |          |             |             | keytrustee=CTc/keytrustee
 template1  | keytrustee | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/keytrustee            +
            |            |          |             |             | keytrustee=CTc/keytrustee
(4 rows)

keytrustee=# \c postgres;
You are now connected to database "postgres" as user "keytrustee".
postgres=# drop database keytrustee;
DROP DATABASE
postgres=# create database keytrustee;
CREATE DATABASE
postgres=# \q
$ sudo -u keytrustee /opt/cloudera/parcels/KEYTRUSTEE_SERVER/PG_DB/opt/postgres/9.3/bin/psql -p 11381 -f /var/lib/keytrustee/kts_bak_kts01_example_com_2016-02-10_11-14-37.sql

For package-based installations:

$ su - keytrustee
$ psql -p 11381
psql (9.3.6)
Type "help" for help.

keytrustee=# \list
                                     List of databases
    Name    |   Owner    | Encoding |   Collate   |    Ctype    |     Access privileges
------------+------------+----------+-------------+-------------+---------------------------
 keytrustee | keytrustee | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 postgres   | keytrustee | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 template0  | keytrustee | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/keytrustee            +
            |            |          |             |             | keytrustee=CTc/keytrustee
 template1  | keytrustee | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/keytrustee            +
            |            |          |             |             | keytrustee=CTc/keytrustee
(4 rows)

keytrustee=# \c postgres;
You are now connected to database "postgres" as user "keytrustee".
postgres=# drop database keytrustee;
DROP DATABASE
postgres=# create database keytrustee;
CREATE DATABASE
postgres=# \q
$ sudo -u keytrustee psql -p 11381 -f /var/lib/keytrustee/kts_bak_kts01_example_com_2016-02-10_11-14-37.sql

Restart Key Trustee Server.
- Using Cloudera Manager: Key Trustee Server service > Actions > Restart
- Using the Command Line: Run the following command on the Key Trustee Server hosts:
```
sudo service keytrusteed restart      #RHEL 6-compatible
sudo systemctl restart keytrusteed    #RHEL 7-compatible
```
Restart the Key Trustee KMS service in Cloudera Manager (Key Trustee KMS service > Actions > Restart).

Restore Active Key Trustee Server from Passive Key Trustee Server

If the Active Key Trustee Server fails, and you do not have a backup, you can restore it from the Passive Key Trustee Server using the following procedure. You can also use this procedure if you need to restore keys that were successfully written to the Passive Key Trustee Server, but are not included in the most recent backup.

The following procedure assumes you have installed Key Trustee Server on the replacement host and (if you are using Cloudera Manager) added the Key Trustee Server service.

Copy the Key Trustee Server database from the Passive Key Trustee Server host to the new Active Key Trustee Server host. Run the following command on the Passive Key Trustee Server host:
```
sudo rsync --exclude recovery.conf -a /var/lib/keytrustee/db root@kts01.example.com:/var/lib/keytrustee/
```
Replace kts01.example.com with the hostname of the new Active Key Trustee Server.
Make sure that the recovery.conf file did not get copied to the Active Key Trustee Server (for example, if there was a typo in your rsync command). Run the following command on the Active Key Trustee Server host:
```
sudo ls -l /var/lib/keytrustee/db/recovery.conf
```
If the file exists on the Active Key Trustee Server host, delete it. Make sure you are on the Active Key Trustee Server host before deleting the file. Do not delete the recovery.conf file on the Passive Key Trustee Server host.
Copy the configuration directory from the Passive Key Trustee Server host to the new Active Key Trustee Server host. Run the following command on the Passive Key Trustee Server host:
```
sudo rsync --exclude .ssl --exclude '*.pid' -a /var/lib/keytrustee/.keytrustee root@kts01.example.com:/var/lib/keytrustee/
```
Replace kts01.example.com with the hostname of the new Active Key Trustee Server.
Create the logs directory and make sure it is owned by the keytrustee user and group:
```
sudo mkdir /var/lib/keytrustee/logs
sudo chown keytrustee:keytrustee /var/lib/keytrustee/logs
```
(Cloudera Manager only) Generate the Key Trustee Server keyring: Key Trustee Server service > Actions > Generate Key Trustee Server Keyring
Set up the database on the Active Key Trustee Server host.
- Using Cloudera Manager: Key Trustee Server service > Actions > Set Up Key Trustee Server Database
- Using the Command Line:
```
sudo ktadmin --confdir /var/lib/keytrustee db --port 11381 --pg-rootdir /var/lib/keytrustee/db --bootstrap --slave kts02.example.com
```
  Replace kts02.example.com with the hostname of the Passive Key Trustee Server.
Start the database.
- Using Cloudera Manager: Key Trustee Server service > Instances > Active Database > Actions > Start this Active Database
- Using the Command Line: Run the following command on the Active Key Trustee Server host:
```
sudo ktadmin --confdir /var/lib/keytrustee db --port 11381 --pg-rootdir /var/lib/keytrustee/db --bootstrap --slave kts02.example.com
```
  Replace kts02.example.com with the hostname of the Passive Key Trustee Server.
Enable synchronous replication.
- Using Cloudera Manager: Key Trustee Server service > Actions > Setup Enable Synchronous Replication in HA mode
- Using the Command Line: Run the following command on the Active Key Trustee Server host:
```
sudo ktadmin --confdir /var/lib/keytrustee enable-synchronous-replication
```
Restart the Active Key Trustee Server.
- Using Cloudera Manager: Key Trustee Server service > Actions > Restart
- Using the Command Line: Run the following command on the Active Key Trustee Server host:
```
sudo service keytrusteed restart      #RHEL 6-compatible
sudo systemctl restart keytrusteed    #RHEL 7-compatible
```
Restart the Key Trustee KMS service in Cloudera Manager (Key Trustee KMS service > Actions > Restart).