Configuring Kafka Security

The following steps describe how to configure Kafka to use Sentry authorization. These steps assume you have installed Kafka and Sentry on your cluster.

For more information, see Installing or Upgrading Kafka and Sentry Installation.

To configure Sentry authentication for Kafka:

1. Go to Kafka > Configuration.
2. Select the checkbox Enable Kerberos Authentication.
3. Select a Sentry service in the Kafka service configuration.
4. Add Super users. Super users can perform any action on any resource in the Kafka cluster. The kafka user is added as a super user by default. Super user requests are authorized without going through Sentry, which provides enhanced performance.
5. Select the checkbox Enable Sentry Privileges Caching to enhance performance.

Authorizable resources are resources or entities in a Kafka cluster that require special permissions for a user to be able to perform actions on them. Kafka has four authorizable resources.

• Cluster, which controls who can perform cluster-level operations such as creating or deleting a topic. This can only have one value, kafka-cluster, as one Kafka cluster cannot have more than one cluster resource.
• Topic, which controls who can perform topic-level operations such as producing and consuming topics. Its value must match exactly the topic name in the Kafka cluster. Wildcards ('*') cannot be used when specifying the topic.
• Consumergroup, which controls who can perform consumergroup-level operations such as joining or describing a consumergroup. Its value must exactly match the group.id of a consumergroup.
• Host, which controls from where specific operations can be performed. Think of this as a way to achieve IP filtering in Kafka. You can set the value of this resource to the wildcard (*), which represents all hosts.

You can perform multiple actions on each resource. The following operations are supported by Kafka, though not all actions are valid on all resources.

• ALL, this is a wildcard action, and represents all possible actions on a resource.
• read
• write
• create
• delete
• alter
• describe
• clusteraction

Privileges define what actions are allowed on a resource. A privilege is represented as a string in Sentry. The following rules apply to a valid privilege.

• Can have at most one Host resource. If you do not specify a Host resource in your privilege string, Host=* is assumed.
• Must have exactly one non-Host resource.
• Must have exactly one action specified at the end of the privilege string.

For example, the following are valid privilege strings:

Host=*->Topic=myTopic->action=ALL
Topic=test->action=ALL

The following examples grant privileges to the role test, so that users in testGroup can create a topic named testTopic and produce to it.

However, before you can assign the test role, you must first create it. To create the test role:

$kafka-sentry -cr -r test

To confirm that the role was created, list the roles:

$ kafka-sentry -lr

If Sentry privileges caching is enabled, as recommended, the new privileges you assign take some time to appear in the system. The time is the time-to-live interval of the Sentry privileges cache, which is set using sentry.kafka.caching.ttl.ms. By default, this interval is 30 seconds. For test clusters, it is beneficial to have changes appear within the system as fast as possible, therefore, Cloudera recommends that you either use a lower time interval, or disable caching with sentry.kafka.caching.enable.

• Allow users in testGroup to create a new topic from localhost.

  $ kafka-sentry -gpr -r test -p "Host=127.0.0.1->Cluster=kafka-cluster->action=create"   

• Allow users in testGroup to describe testTopic from localhost, which the user creates and uses.

  $ kafka-sentry -gpr -r test -p "Host=127.0.0.1->Topic=testTopic->action=describe"

• Allow users in testGroup to write to testTopic from localhost, which allows users to produce to testTopic.

  $ kafka-sentry -gpr -r test -p "Host=127.0.0.1->Topic=testTopic->action=write"

• Create testTopic.

  $ kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 \
    --partitions 1 --topic testTopic
  $ kafka-topics --list --zookeeper localhost:2181 testTopic

• Assign the test role to the group testGroup:

  kafka-sentry -arg -r test -g testGroup

• To verify that the test role is part of the group testGroup:

  kafka-sentry -lr -g testGroup

• Produce to testTopic. Note that you have to pass a configuration file, producer.properties, with information on JAAS configuration and other Kerberos authentication related information. See SASL Configuration for Kafka Clients.

  $ kafka-console-producer --broker-list localhost:9092 --topic testTopic \
    --producer.config producer.properties
  This is a message
  This is another message

• Allow users in testGroup to describe a consumer group, testconsumergroup, that it will start and join.

  $ kafka-sentry -gpr -r test -p "Host=127.0.0.1->Consumergroup=testconsumergroup->action=describe"

• Allow users in testGroup to read from a consumer group, testconsumergroup, that it will start and join.

  $ kafka-sentry -gpr -r test -p "Host=127.0.0.1->Topic=testTopic->action=read"

• Allow users in testGroup to read from testTopic from localhost and to consume from testTopic.

  $ kafka-sentry -gpr -r test -p "Host=127.0.0.1->Consumergroup=testconsumergroup->action=read"

• Consume from testTopic. Note that you have to pass a configuration file, consumer.properties, with information on JAAS configuration and other Kerberos authentication related information. The configuration file must also specify group.id as testconsumergroup.

  $ kafka-console-consumer --zookeeper localhost:2181 --topic testTopic --from-beginning --consumer.config consumer.properties
  This is a message
  This is another message

If Kafka requests are failing due to authorization, the following steps can provide insight into the error:

• Make sure you are kinit'd as a user who has privileges to perform an operation.
• Identify which broker is hosting leader of the partition you are trying to produce to or consume from, as this leader is going to authorize your request against Sentry. One easy way of debugging is to just have one Kafka broker. Change log level of the Kafka broker to debug and restart the broker.
• Run the Kafka client or Kafka CLI with required arguments and capture the Kafka log, which should be something like /var/log/kafka/kafka-broker-<HOST_ID>.log on kafka broker's host.
• There will be many Jetty logs, and filtering that out usually helps in reducing noise. Look for log messages from org.apache.sentry.
• Look for following information in the filtered logs:
  • Groups the user Kafka client or CLI is running as.
  • Required privileges for the operation.
  • Retrieved privileges from Sentry.
  • Required and retrieved privileges comparison result.

This log information can provide insight into which privilege is not assigned to a user, causing a particular operation to fail.