Managing secrets using the REST API
Learn how you can use the Kafka Connect Secrets Storage feature to manage secrets in connector configurations using the Kafka Connect REST API.
Mark a configuration value as a secret in a connector configuration
To mark a property value in a connector configuration as a secret, you must:
- Add the property to the configuration.
- Add the
secret.properties
property. The value of this property is a comma separated list of the property keys that you want to mark as secrets.
For example, assume that you have two sensitive properties in your configuration that you
want to mark as secrets, password.for.connector
and
some.confidential.config
.
{
“name” = “Custom”,
“connector.class” = “my.custom.secure.connector”,
“secret.properties” = “password.for.connector,some.confidential.config”,
“topics” = “...”,
“password.for.connector” = “mypassword”,
“some.confidential.config” = “sensitive information”
}
If the secret.properties
is present in the configuration, the Kafka
Connect Secrets Storage feature gets enabled. Once the connector configuration is submitted,
the feature processes the configuration and makes a number of changes.
{
“name” = “Custom”,
“connector.class” = “my.custom.secure.connector”,
“secret.properties” = “password.for.connector,some.confidential.config”,
“secret.bundle.id” = “18a1b3a6-1e2f”,
“topics” = “...”,
“password.for.connector” = “${secret:Custom/18a1b3a6-1e2f:password.for.connector}”,
“some.confidential.config” = “${secret:Custom/18a1b3a6-1e2f:some.confidential.config}”
}
Notice the following changes:
- A
secret.bundle.id
property is added. The bundle is the set of secrets of a secure connector.The bundle is identified by its bundle ID, which is maintained by the Kafka Connect Secrets Storage feature. If a secret is added, deleted, or edited, a new bundle is created with a new bundle ID.
- The
password.for.connector
andsome.confidential.config
properties had their values replaced with secret references.These references are placeholder values consisting of the connector name, bundle ID, and property key. If a reference like this is present in the configuration, it means that the value for that property is securely stored in an internal Kafka topic (secrets topic). The Kafka Connect Secrets Storage feature resolves these references and substitutes them with the original values stored in the secrets topic when the connectors are used.
Update a connector configuration with a new secret
If you want to update an already existing configuration with a new secret property, you
must add the new property to the configuration as well as the
secret.properties
property. For example, assume that you have a
configuration where password.for.connector
and
some.confidential.config
are already marked as secrets. Now, you want to
add the new.secret.config
property to the configuration and mark it as a
secret as well.
{
“name” = “Custom”,
“connector.class” = “my.custom.secure.connector”,
“secret.properties” = “password.for.connector,some.confidential.config,new.secret.config”,
“secret.bundle.id” = “18a1b3a6-1e2f”,
“topics” = “...”,
“password.for.connector” = “${secret:Custom/18a1b3a6-1e2f:password.for.connector}”,
“some.confidential.config” = “${secret:Custom/18a1b3a6-1e2f:some.confidential.config}”,
“new.secret.config” = “[***A NEW SECRET VALUE***]”
}
Once submitted and processed, the configuration will look like the following example:
{
“name” = “Custom”,
“connector.class” = “my.custom.secure.connector”,
“secret.properties” = “password.for.connector,some.confidential.config,new.secret.config”,
“secret.bundle.id” = “c8a753d6-8242”,
“topics” = “...”,
“password.for.connector” = “${secret:Custom/c8a753d6-8242:password.for.connector}”,
“some.confidential.config” = “${secret:Custom/c8a753d6-8242:some.confidential.config}”,
“new.secret.config” = “${secret:Custom/c8a753d6-8242:new.secret.config}”
}
Notice the following changes:
- The
secret.bundle.id
value has changed.This is done because adding a new secret creates a new set of secrets resulting in a new bundle being created.
- The values (secret references) of
password.for.connector
andsome.confidential.config
are updated with the new bundle ID. - The value of
new.secret.config
is replaced by a secret reference.This means that the value is now securely stored in the secrets topic.
Update a connector configuration by changing a secret
Updating already existing secret values can be done by updating the appropriate property
values. For example, assume that you have three properties marked as secrets,
password.for.connector
, some.confidential.config
, and
new.secret.config
. You want to update the value of
new.secret.config
.
{
“name” = “Custom”,
“connector.class” = “my.custom.secure.connector”,
“secret.properties” = “password.for.connector,some.confidential.config,new.secret.config”,
“secret.bundle.id” = “c8a753d6-8242”,
“topics” = “...”,
“password.for.connector” = “${secret:Custom/c8a753d6-8242:password.for.connector}”,
“some.confidential.config” = “${secret:Custom/c8a753d6-8242:some.confidential.config}”,
“new.secret.config” = “[***CHANGED VALUE***]”
}
Once submitted and processed, the configuration will look like the following example:
{
“name” = “Custom”,
“connector.class” = “my.custom.secure.connector”,
“secret.properties” = “password.for.connector,some.confidential.config,new.secret.config”,
“secret.bundle.id” = “14b2f85f-1dcb”,
“topics” = “...”,
“password.for.connector” = “${secret:Custom/14b2f85f-1dcb:password.for.connector}”,
“some.confidential.config” = “${secret:Custom/14b2f85f-1dcb:some.confidential.config}”,
“new.secret.config” = “${secret:Custom/14b2f85f-1dcb:new.secret.config}”
}
Notice the following changes:
- The
secret.bundle.id
value has changed.This is done because deleting a secret creates a new set of secrets resulting in a new bundle being created. In this case the deleted secret is removed from the bundle.
- The values (secret references) of
password.for.connector
andsome.confidential.config
are updated with the new bundle ID.This is done because updating a secret creates a new set of secrets resulting in a new bundle being created.
- The value of
new.secret.config
is replaced by a secret reference.This means that the new value is now securely stored in the secrets topic.
Remove an existing secret from an existing configuration
Secrets can be removed from a configuration by updating the configuration and deleting the
unnecessary properties from both the configuration and secret.properties
.
For example, assume you have three properties marked as secrets,
password.for.connector
, some.confidential.config
, and
new.secret.config
.
{
“name” = “Custom”,
“connector.class” = “my.custom.secure.connector”,
“secret.properties” = “password.for.connector,some.confidential.config,new.secret.config”,
“secret.bundle.id” = “14b2f85f-1dcb”,
“topics” = “...”,
“password.for.connector” = “${secret:Custom/14b2f85f-1dcb:password.for.connector}”,
“some.confidential.config” = “${secret:Custom/14b2f85f-1dcb:some.confidential.config}”,
“new.secret.config” = “${secret:Custom/14b2f85f-1dcb:new.secret.config}”
}
Assume you want to delete some.confidential.config
. To do this, you need
to delete the property from the configuration and remove the property key from
secret.properties
.
{
“name” = “Custom”,
“connector.class” = “my.custom.secure.connector”,
“secret.properties” = “password.for.connector,new.secret.config”,
“secret.bundle.id” = “14b2f85f-1dcb”,
“topics” = “...”,
“password.for.connector” = “${secret:Custom/14b2f85f-1dcb:password.for.connector}”,
“new.secret.config” = “${secret:Custom/14b2f85f-1dcb:new.secret.config}”
}
Once submitted and processed, the configuration will look like the following example:
{
“name” = “Custom”,
“connector.class” = “my.custom.secure.connector”,
“secret.properties” = “password.for.connector,new.secret.config”,
“secret.bundle.id” = “15641fca-483b”,
“topics” = “...”,
“password.for.connector” = “${secret:Custom/15641fca-483b:password.for.connector}”,
“new.secret.config” = “${secret:Custom/15641fca-483b:new.secret.config}”
}
Notice the following changes:
- The
secret.bundle.id
value has changed.This is done because deleting a secret creates a new set of secrets resulting in a new bundle being created. In this case the deleted secret is removed from the bundle.
-
The values (secret references) of
password.for.connector
andsome.confidential.config
are updated with the new bundle ID.
Remove all secrets
To remove all secrets from a configuration, you must remove all properties that were marked
as secrets and also remove the secret.properties
property. However, you
must leave secret.bundle.id
intact. For example, assume you had a single
property marked as a secret, password.for.connector
.
{
“name” = “Custom”,
“connector.class” = “my.custom.secure.connector”,
“secret.properties” = “password.for.connector,new.secret.config”,
“secret.bundle.id” = “c8a753d6-8242”,
“topics” = “...”,
“password.for.connector”= “${secret:Custom/c8a753d6-8242:password.for.connector}
}
In a case like this, you need to remove password.for.connector
and
secret.properties
from the configuration. However, you must preserve
secret.bundle.id.
The reason why you need to leave this property intact
is because the encrypted values for the secret properties must be deleted from the internal
secrets topic. The Kafka Connect Secrets Storage feature can only do this if it has a bundle
ID for reference.
{
“name” = “Custom”,
“connector.class” = “my.custom.secure.connector”,
“secret.bundle.id” = “c8a753d6-8242”,
“topics” = “...”
}
Once the configuration is submitted and processed, secret.bundle.id
is
removed. The connector configuration at this point is considered unsecured.
{
“name” = “Custom”,
“connector.class” = “my.custom.secure.connector”,
“topics” = “...”
}