Managing secrets using the REST API

Learn how you can use the Kafka Connect Secrets Storage feature to manage secrets in connector configurations using the Kafka Connect REST API.

You can use the Kafka Connect Secrets Storage feature with the Kafka Connect REST API. The following sections walk you through how you can mark properties as secrets and how you can manage already existing secrets.

Mark a configuration value as a secret in a connector configuration

To mark a property value in a connector configuration as a secret, you must:

Add the property to the configuration.
Add the secret.properties property. The value of this property is a comma separated list of the property keys that you want to mark as secrets.

For example, assume that you have two sensitive properties in your configuration that you want to mark as secrets, password.for.connector and some.confidential.config.

{
“name” = “Custom”,
“connector.class” = “my.custom.secure.connector”,
“secret.properties” = “password.for.connector,some.confidential.config”,
“topics” = “...”,
“password.for.connector” = “mypassword”,
“some.confidential.config” = “sensitive information”
}

If the secret.properties is present in the configuration, the Kafka Connect Secrets Storage feature gets enabled. Once the connector configuration is submitted, the feature processes the configuration and makes a number of changes.

{
“name” = “Custom”,
“connector.class” = “my.custom.secure.connector”,
“secret.properties” = “password.for.connector,some.confidential.config”,
“secret.bundle.id” = “18a1b3a6-1e2f”,
“topics” = “...”,
“password.for.connector” = “${secret:Custom/18a1b3a6-1e2f:password.for.connector}”,
“some.confidential.config” = “${secret:Custom/18a1b3a6-1e2f:some.confidential.config}”
}

Notice the following changes:

A secret.bundle.id property is added. The bundle is the set of secrets of a secure connector.
The bundle is identified by its bundle ID, which is maintained by the Kafka Connect Secrets Storage feature. If a secret is added, deleted, or edited, a new bundle is created with a new bundle ID.
The password.for.connector and some.confidential.config properties had their values replaced with secret references.
These references are placeholder values consisting of the connector name, bundle ID, and property key. If a reference like this is present in the configuration, it means that the value for that property is securely stored in an internal Kafka topic (secrets topic). The Kafka Connect Secrets Storage feature resolves these references and substitutes them with the original values stored in the secrets topic when the connectors are used.

Update a connector configuration with a new secret

If you want to update an already existing configuration with a new secret property, you must add the new property to the configuration as well as the secret.properties property. For example, assume that you have a configuration where password.for.connector and some.confidential.config are already marked as secrets. Now, you want to add the new.secret.config property to the configuration and mark it as a secret as well.

{
“name” = “Custom”,
“connector.class” = “my.custom.secure.connector”,
“secret.properties” = “password.for.connector,some.confidential.config,new.secret.config”,
“secret.bundle.id” = “18a1b3a6-1e2f”,
“topics” = “...”,
“password.for.connector” = “${secret:Custom/18a1b3a6-1e2f:password.for.connector}”,
“some.confidential.config” = “${secret:Custom/18a1b3a6-1e2f:some.confidential.config}”,
“new.secret.config” = “[***A NEW SECRET VALUE***]”
}

Once submitted and processed, the configuration will look like the following example:

{
“name” = “Custom”,
“connector.class” = “my.custom.secure.connector”,
“secret.properties” = “password.for.connector,some.confidential.config,new.secret.config”,
“secret.bundle.id” = “c8a753d6-8242”,
“topics” = “...”,
“password.for.connector” = “${secret:Custom/c8a753d6-8242:password.for.connector}”,
“some.confidential.config” = “${secret:Custom/c8a753d6-8242:some.confidential.config}”,
“new.secret.config” = “${secret:Custom/c8a753d6-8242:new.secret.config}”
}

Notice the following changes:

The secret.bundle.id value has changed.
This is done because adding a new secret creates a new set of secrets resulting in a new bundle being created.
The values (secret references) of password.for.connector and some.confidential.config are updated with the new bundle ID.
The value of new.secret.config is replaced by a secret reference.
This means that the value is now securely stored in the secrets topic.

Update a connector configuration by changing a secret

Updating already existing secret values can be done by updating the appropriate property values. For example, assume that you have three properties marked as secrets, password.for.connector, some.confidential.config, and new.secret.config. You want to update the value of new.secret.config.

{
“name” = “Custom”,
“connector.class” = “my.custom.secure.connector”,
“secret.properties” = “password.for.connector,some.confidential.config,new.secret.config”,
“secret.bundle.id” = “c8a753d6-8242”,
“topics” = “...”,
“password.for.connector” = “${secret:Custom/c8a753d6-8242:password.for.connector}”,
“some.confidential.config” = “${secret:Custom/c8a753d6-8242:some.confidential.config}”,
“new.secret.config” = “[***CHANGED VALUE***]”
}

Once submitted and processed, the configuration will look like the following example:

{
“name” = “Custom”,
“connector.class” = “my.custom.secure.connector”,
“secret.properties” = “password.for.connector,some.confidential.config,new.secret.config”,
“secret.bundle.id” = “14b2f85f-1dcb”,
“topics” = “...”,
“password.for.connector” = “${secret:Custom/14b2f85f-1dcb:password.for.connector}”,
“some.confidential.config” = “${secret:Custom/14b2f85f-1dcb:some.confidential.config}”,
“new.secret.config” = “${secret:Custom/14b2f85f-1dcb:new.secret.config}”
}

Notice the following changes:

The secret.bundle.id value has changed.
This is done because deleting a secret creates a new set of secrets resulting in a new bundle being created. In this case the deleted secret is removed from the bundle.
The values (secret references) of password.for.connector and some.confidential.config are updated with the new bundle ID.
This is done because updating a secret creates a new set of secrets resulting in a new bundle being created.
The value of new.secret.config is replaced by a secret reference.
This means that the new value is now securely stored in the secrets topic.

Remove an existing secret from an existing configuration

Secrets can be removed from a configuration by updating the configuration and deleting the unnecessary properties from both the configuration and secret.properties. For example, assume you have three properties marked as secrets, password.for.connector, some.confidential.config, and new.secret.config.

{
“name” = “Custom”,
“connector.class” = “my.custom.secure.connector”,
“secret.properties” = “password.for.connector,some.confidential.config,new.secret.config”,
“secret.bundle.id” = “14b2f85f-1dcb”,
“topics” = “...”,
“password.for.connector” = “${secret:Custom/14b2f85f-1dcb:password.for.connector}”,
“some.confidential.config” = “${secret:Custom/14b2f85f-1dcb:some.confidential.config}”,
“new.secret.config” = “${secret:Custom/14b2f85f-1dcb:new.secret.config}”
}

Assume you want to delete some.confidential.config. To do this, you need to delete the property from the configuration and remove the property key from secret.properties.

{
“name” = “Custom”,
“connector.class” = “my.custom.secure.connector”,
“secret.properties” = “password.for.connector,new.secret.config”,
“secret.bundle.id” = “14b2f85f-1dcb”,
“topics” = “...”,
“password.for.connector” = “${secret:Custom/14b2f85f-1dcb:password.for.connector}”,
“new.secret.config” = “${secret:Custom/14b2f85f-1dcb:new.secret.config}”
}

Once submitted and processed, the configuration will look like the following example:

{
“name” = “Custom”,
“connector.class” = “my.custom.secure.connector”,
“secret.properties” = “password.for.connector,new.secret.config”,
“secret.bundle.id” = “15641fca-483b”,
“topics” = “...”,
“password.for.connector” = “${secret:Custom/15641fca-483b:password.for.connector}”,
“new.secret.config” = “${secret:Custom/15641fca-483b:new.secret.config}”
}

Notice the following changes:

The secret.bundle.id value has changed.
This is done because deleting a secret creates a new set of secrets resulting in a new bundle being created. In this case the deleted secret is removed from the bundle.
The values (secret references) of password.for.connector and some.confidential.config are updated with the new bundle ID.

Remove all secrets

To remove all secrets from a configuration, you must remove all properties that were marked as secrets and also remove the secret.properties property. However, you must leave secret.bundle.id intact. For example, assume you had a single property marked as a secret, password.for.connector.

{
“name” = “Custom”,
“connector.class” = “my.custom.secure.connector”,
“secret.properties” = “password.for.connector,new.secret.config”,
“secret.bundle.id” = “c8a753d6-8242”,
“topics” = “...”,
“password.for.connector”= “${secret:Custom/c8a753d6-8242:password.for.connector}
}

In a case like this, you need to remove password.for.connector and secret.properties from the configuration. However, you must preserve secret.bundle.id. The reason why you need to leave this property intact is because the encrypted values for the secret properties must be deleted from the internal secrets topic. The Kafka Connect Secrets Storage feature can only do this if it has a bundle ID for reference.

{
“name” = “Custom”,
“connector.class” = “my.custom.secure.connector”,
“secret.bundle.id” = “c8a753d6-8242”,
“topics” = “...”
}

Once the configuration is submitted and processed, secret.bundle.id is removed. The connector configuration at this point is considered unsecured.

{
“name” = “Custom”,
“connector.class” = “my.custom.secure.connector”,
“topics” = “...”
}