Kafka Delegation Tokens

Delegation tokens were introduced as a lightweight authentication method to complement existing SASL authentication. Kafka is designed to support a high number of clients. However, using Kerberos authentication might be difficult in some environments due to the following reasons:

  • With Kerberos authentication, all clients need access to a keytab or a TGT. Securely distributing the keytabs requires a lot of effort and careful administration. When the TGT is compromised, it has a high blast radius, especially when the same keytabs are used to access multiple services.

  • With Kerberos, client authentication is centralized, and the high number of clients can put a high load on the KDC (Key Distribution Center), resulting in a bottleneck.

Many Hadoop components use delegation tokens to mitigate these problems:

  • Delegation tokens allow these components to secure distributed workloads with low administrative overhead

  • It is not required to distribute a Kerberos TGT or keytab, which, if compromised, may grant access to all services.

  • A Delegation token is strictly tied to its associated service causing less damage if exposed.

  • Delegation tokens make credential renewal more lightweight. This is because the renewal is designed in such a way that only the renewer and the service are involved in the renewal process. The token itself remains the same, so parties already using the token do not have to be updated.

Delegation Token Basics

Kafka delegation tokens were modeled after Hadoop delegation tokens, and many of their mechanism are the same or very similar. However, this does not mean that they are interchangeable.

Delegation tokens are generated and verified following the HMAC mechanism. There are two basic parts of information in a delegation Token:
  • Public part (tokenID)
  • Private part (HMAC value)
The following steps describe the basics of how delegation tokens are used:

  1. The user initially authenticates with the Kafka cluster via SASL, and obtains a delegation token using either the AdminClient APIs or the kafka-delegation-token tool. The principal that created the delegation token is its owner.

  2. The delegation token details are securely passed to Kafka clients. This can be achieved by sending the token data over an SSL/TLS encrypted connection or writing them to a secure shared storage.

  3. Instead of using Kerberos, the Kafka client uses the delegation tokens for subsequent authentication with the brokers. See The Client authentication using delegation tokens.

  4. The token is valid for a certain time period, but it can be:

    Renewed

    A delegation token can be renewed multiple times up until its maximum life before it expires. The token can be renewed by the owner or any other principals the owner sets as “renewer” at time of creation.

    Revoked
    A delegation token can be revoked ahead of its expiry time.

Broker Configuration Settings

Certain delegation token properties can be configured on a service level in Cloudera Manager. Delegation Token Broker Configuration Settingsgives an overview of these properties.
Delegation Token Broker Configuration Settings
Name Description Type Default
delegation.token.enable Enables authentication with delegation tokens for this Kafka service. When enabled, a secure password is automatically generated and used as the "delegation.token.master.key" property for Kafka Brokers. Only allowed if Kerberos authentication is enabled. boolean false
delegation.token.expiry.time.ms The expiry time of a delegation token. A delegation token can be renewed before its expiry time to extend its validity up to its maximum lifetime. If it is not renewed, it will expire even if it has time remaining from its maximum lifetime. time 1 day
delegation.token.max.lifetime.ms The maximum amount of time that a delegation token can be valid for. time 7 days