Configuring Apache Kafka Security

The following steps describe how to configure Kafka to use Sentry authorization. These steps assume you have installed Kafka and Sentry on your cluster.

For more information, see Installing or Upgrading CDK Powered By Apache Kafka® and Installing and Upgrading the Sentry Service.

To configure Sentry authentication for Kafka:

1. Go to Kafka > Configuration.
2. Select the checkbox Enable Kerberos Authentication.
3. Select a Sentry service in the Kafka service configuration.
4. Add Super users. Super users can perform any action on any resource in the Kafka cluster. The kafka user is added as a super user by default. Super user requests are authorized without going through Sentry, which provides enhanced performance.
5. Select the checkbox Enable Sentry Privileges Caching to enhance performance.

Authorizable resources are resources or entities in a Kafka cluster that require special permissions for a user to be able to perform actions on them. Kafka has four authorizable resources.

• Cluster, which controls who can perform cluster-level operations such as creating or deleting a topic. This can only have one value, kafka-cluster, as one Kafka cluster cannot have more than one cluster resource.
• Topic, which controls who can perform topic-level operations such as producing and consuming topics. Its value must match exactly the topic name in the Kafka cluster. With CDK 3.1.0 and CDH 5.14.2 and later, wildcards (*) can be used to refer to any topic in the privilege.

• Consumergroup, which controls who can perform consumergroup-level operations such as joining or describing a consumergroup. Its value must exactly match the group.id of a consumergroup. With CDK 3.1.0 and CDH 5.14.2 and later, you can use a wildcard (*) to refer to any consumer groups in the privilege. This is useful when used with Spark Streaming, where a generated group.id may be needed.

• Host, which controls from where specific operations can be performed. Think of this as a way to achieve IP filtering in Kafka. You can set the value of this resource to the wildcard (*), which represents all hosts.

You can perform multiple actions on each resource. The following operations are supported by Kafka, though not all actions are valid on all resources.

• ALL, this is a wildcard action, and represents all possible actions on a resource.
• read
• write
• create
• delete
• alter
• describe
• clusteraction

Privileges define what actions are allowed on a resource. A privilege is represented as a string in Sentry. The following rules apply to a valid privilege.

• Can have at most one Host resource. If you do not specify a Host resource in your privilege string, Host=* is assumed.
• Must have exactly one non-Host resource.
• Must have exactly one action specified at the end of the privilege string.

For example, the following are valid privilege strings:

Host=*->Topic=myTopic->action=ALL
Topic=test->action=ALL

The following examples grant privileges to the role test, so that users in testGroup can create a topic named testTopic and produce to it.

The user executing these commands must be added to the Sentry parameter sentry.service.allow.connect and also be a member of a group defined in sentry.service.admin.group.

Before you can assign the test role, you must first create it. To create the test role:

$kafka-sentry -cr -r test

To confirm that the role was created, list the roles:

$ kafka-sentry -lr

If Sentry privileges caching is enabled, as recommended, the new privileges you assign take some time to appear in the system. The time is the time-to-live interval of the Sentry privileges cache, which is set using sentry.kafka.caching.ttl.ms. By default, this interval is set to 30 seconds. For test clusters, it is beneficial to have changes appear within the system as fast as possible, therefore, Cloudera recommends that you either use a lower time interval, or disable caching with sentry.kafka.caching.enable.

• Allow users in testGroup to write to testTopic from localhost, which allows users to produce to testTopic. They need both write and describe permissions.

  $ kafka-sentry -gpr -r test -p "Host=127.0.0.1->Topic=testTopic->action=write"
  $ kafka-sentry -gpr -r test -p "Host=127.0.0.1->Topic=testTopic->action=describe"

• Assign the test role to the group testGroup:

  kafka-sentry -arg -r test -g testGroup

• Verify that the test role is part of the group testGroup:

  kafka-sentry -lr -g testGroup

• Create testTopic.

  $ kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 \
    --partitions 1 --topic testTopic
  $ kafka-topics --list --zookeeper localhost:2181 testTopic

• Produce to testTopic. Note that you have to pass a configuration file, producer.properties, with information on JAAS configuration and other Kerberos authentication related information. See SASL Configuration for Kafka Clients.

  $ kafka-console-producer --broker-list localhost:9092 --topic testTopic \
    --producer.config producer.properties
  This is a message
  This is another message

• Grant the create privilege to the test role.

  $ kafka-sentry -gpr -r test -p "Host=127.0.0.1->Cluster=kafka-cluster->action=create"   

• Allow users in testGroup to describe testTopic from localhost, which the user creates and uses.

  $ kafka-sentry -gpr -r test -p "Host=127.0.0.1->Topic=testTopic->action=describe"

• Grant the describe privilege to the test role.

  $ kafka-sentry -gpr -r test -p "Host=127.0.0.1->Consumergroup=testconsumergroup->action=describe"

• Allow users in testGroup to read from a consumer group, testconsumergroup, that it will start and join.

  $ kafka-sentry -gpr -r test -p "Host=127.0.0.1->Consumergroup=testconsumergroup->action=read"

• Allow users in testGroup to read from testTopic from localhost and to consume from testTopic.

  $ kafka-sentry -gpr -r test -p "Host=127.0.0.1->Topic=testTopic->action=read"

• Consume from testTopic. Note that you have to pass a configuration file, consumer.properties, with information on JAAS configuration and other Kerberos authentication related information. The configuration file must also specify group.id as testconsumergroup.

  kafka-console-consumer --new-consumer --topic test1 --from-beginning --bootstrap-server <anybroker>:9092 --consumer.config consumer.properties
  This is a message
  This is another message

If Kafka requests are failing due to authorization, the following steps can provide insight into the error:

• Make sure you are kinit'd as a user who has privileges to perform an operation.
• Identify which broker is hosting leader of the partition you are trying to produce to or consume from, as this leader is going to authorize your request against Sentry. One easy way of debugging is to just have one Kafka broker. Change log level of the Kafka broker to debug and restart the broker.
• Run the Kafka client or Kafka CLI with required arguments and capture the Kafka log, which should be something like /var/log/kafka/kafka-broker-<HOST_ID>.log on kafka broker's host.
• There will be many Jetty logs, and filtering that out usually helps in reducing noise. Look for log messages from org.apache.sentry.
• Look for following information in the filtered logs:
  • Groups the user Kafka client or CLI is running as.
  • Required privileges for the operation.
  • Retrieved privileges from Sentry.
  • Required and retrieved privileges comparison result.

This log information can provide insight into which privilege is not assigned to a user, causing a particular operation to fail.