Re-encrypting secrets

Using the Kafka Connect Secrets Storage feature places encrypted confidential information in an internal Kafka topic (secrets topic) used by the feature. If you want to change the encryption key used to encrypt the information placed in the topic, all secrets must be re-encrypted and must be migrated to a new topic. This is done using the connect-secret-storage-migration tool that is shipped with Cloudera Runtime.

The connect-secret-storage-migration tool migrates existing secrets from the currently used secrets topic (source) to a newly created one (target). It achieves this by reading through the source topic, decrypting the secrets stored within it using the existing global key, and re-encrypts them with a new, randomly generated encryption key and a new global key that you configure. After encryption is finished, newly encrypted secrets are produced into the target topic. The new encryption key will have a different security context (different global password and salt) than the currently used one.

The tool requires three configuration files to function. These configuration files provide the tool with information regarding the Kafka service as well as the source and target topics. The configuration files must be created manually.

In the configuration files you prepare, you must specify the current global password and PBE salt used by the feature. These values are specified in Cloudera Manager with the Kafka Connect Secrets Storage PBE Salt and Kafka Connect Secrets Storage Global Password properties. The default values for these properties are randomly generated and are hidden, but can be changed at any time. As a result, if you do not know the global password or PBE salt value, you must change both values, note down the values you configured, and restart all Kafka Connect roles before you complete the following steps.

Stop all Kafka Connect roles.
Prepare the configuration files required for the tool.
You need to prepare three different configuration files. A Kafka client configuration file that the tool can use to access the Kafka service. Additionally, two configuration files that contain information regarding the source and target topics. All three files must be deployed on the cluster host where you run the tool.
1. Create a Kafka client configuration file.
  This configuration file must contain all client properties that the tool requires to access the Kafka service. The properties you need to specify in this file depend on the security configuration of the Kafka service. The following tabs contain examples for some of the most commonly used security configurations.
  - SASL_SSL Kerberos
  - SASL_SSL PLAIN
  bootstrap.servers=[***HOST***]:[***PORT***] security.protocol=SASL_SSL sasl.mechanism=GSSAPI sasl.kerberos.service.name=kafka sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true keyTab="[***PATH TO KEYTAB***]" principal="[***KERBEROS PRINCIPAL***]"; ssl.truststore.location=[***PATH TO TRUSTSTORE.JKS***] ssl.truststore.password=[***TRUSTSTORE PASSWORD***]
  bootstrap.servers=[***HOST***]:[***PORT***] security.protocol=SASL_SSL sasl.mechanism=PLAIN sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="[***USERNAME***]" password="[***PASSWORD***]"; ssl.truststore.location=[***PATH TO TRUSTSTORE.JKS***] ssl.truststore.password=[***TRUSTSTORE PASSWORD***]
2. Create the configuration file identifying the source topic and its security context. For example:
```
kafka.connect.secret.storage.topic : [***SOURCE TOPIC NAME***]
kafka.connect.secret.global.password : [***CURRENT GLOBAL PASSWORD***]
kafka.connect.secret.pbe.salt : [***CURRENT PBE SALT***]
```
  Replace [***SOURCE TOPIC NAME***] with the name of the currently used secrets topic. The name of the current topic is specified in Cloudera Manager in the Kafka Connect Secrets Storage Topic Name property.
  
  Replace [***GLOBAL PASSWORD***] with the currently used global password.
  
  Replace [***PBE SALT***] with the currently used PBE salt value.
  
  You can choose to not specify kafka.connect.secret.global.password in the configuration. If the property is not added to the configuration, you are prompted to enter the password when you run the tool.
3. Create the configuration file identifying the target topic and its security context. For example:
```
kafka.connect.secret.storage.topic : [***TARGET TOPIC NAME***]
kafka.connect.secret.global.password : [***NEW GLOBAL PASSWORD***]
kafka.connect.secret.pbe.salt : [***NEW PBE SALT***]
kafka.connect.secret.storage.topic.replication.factor : [***REPLICATION FACTOR***]
kafka.connect.secret.storage.topic.configs.min.insync.replicas : [***IN-SYNC REPLICA NUMBER***]
```
  You can choose to not specify kafka.connect.secret.global.password in the configuration. If the property is not added to the configuration, you are prompted to enter the password when you run the tool.
Run the tool.
You must specify the three configuration files you created as parameters. The order in which you specify the files is fixed. The first file must be the Kafka client configuration, the second must be the file for the source topic, and the third file must be the file for the target topic. For example:
```
connect-secret-storage-migration [***KAFKA CLIENT CONFIGURATION***] [***SOURCE TOPIC CONFIGURATION***] [***TARGET TOPIC CONFIGURAITON***]
```
If prompted, enter the current and new global password.
The tool only prompts you to enter the passwords if they were not specified in the configuration files. The tool does not echo the passwords that you type in the console.
The tool prompts you to ensure that the Kafka Connect cluster (all Connect roles) are stopped. Press Enter to continue.
Wait until the tool is finished with re-encryption.
The tool reports its progress. Upon successful completion, the tool exits. Upon failure, an error message is printed. In such a case, fix any issues that the tool reports and run the tool again.
In Cloudera Manager, select the Kafka service.
Go to Configuration.
Find and configure the following properties:
- Kafka Connect Secrets Storage Topic Name
- Kafka Connect Secrets Storage Topic Replication Factor
- Kafka Connect Secrets Storage Topic Minimum In-Sync Replicas
- Kafka Connect Secrets Storage Global Password
- Kafka Connect Secrets Storage PBE Salt
Add the values you specified in the configuration file identifying the target topic.
Restart all Kafka Connect roles.
Verify that Kafka Connect is working as expected.
This can be done by verifying that all connectors that have properties marked as secrets are running and are in a healthy state. That is, the connectors are resolving all referenced secrets correctly. If you are experiencing issues, you can revert the configuration changes in Cloudera Manager and restart the re-encryption process.
Delete the configuration files you created.
Delete the source secrets topic.

important
Ensure that you verified that Kafka Connect and your connectors are working as expected before deleting the topic. When you delete the topic, the secrets stored within it are lost. As a result, if there are issues with Kafka Connect, you might not be able to revert the cluster to a working state.