Kafka Security Hardening with Zookeeper ACLs

Restricting Access to Kafka Metadata in Zookeeper

Locking down znodes in Zookeeper can be used to protect Kafka metadata against unauthorized access. Direct manipulation of metadata in Zookeeper is not only dangerous for the health of the cluster, but can also serve as an entry point for malicious users to gain elevated access who can then alter the owner or renewer of delegation tokens.

Prerequisites

A secure Kafka cluster with Kerberos authentication enabled is required. For more information see, Enabling Kerberos Authentication.

Steps

Restrict access to Kafka metadata stored in ZooKeeper by completing the following steps:
  1. Enable the use of secure ACLs by setting zookeeper.set.acl configuration parameter to true.
    1. In Cloudera Manager go to the Kafka service.
    2. Select Configuration and find the Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka.properties property.
    3. Add the following line to the Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka.properties property: 
      zookeeper.set.acl=true

  2. Perform a Rolling Restart

    1. Return to the Home page by clicking the Cloudera Manager logo.
    2. Go to the Kafka service and select Actions Rolling Restart.
    3. Check the Restart roles with stale configurations only checkbox and click Rolling restart.
    4. Click Close when the restart has finished.
  3. Pass the JAAS config file location as a JVM parameter through a command line interface. You can do this by setting the value of the KAFKA_OPTS environment variable to -Djava.security.auth.login.config=path/to/jaas.conf. 
    export KAFKA_OPTS="-Djava.security.auth.login.config=path/to/jaas.conf"
  4. Run the zookeeper-security-migration tool with the zookeeper.acl option set to secure. 
    zookeeper-security-migration --zookeeper.connect hostname:port --zookeeper.acl secure

    The tool traverses the corresponding sub-trees changing the ACLs of the znodes

  5. Reset the ACLs on the root node to allow full access.

    Resetting the ACLS on the root node is required because the zookeeper-security-migration tool also changes the ACLs on the root znode. This leads to the failure of the Zookeeper canary tests, which subsequently makes the service display as unhealthy in Cloudera Manager.

    1. Change the JVMFLAGS environment variable to -Djava.security.auth.login.config=path/to/jaas.conf 
      export JVMFLAGS="-Djava.security.auth.login.config=path/to/jaas.conf"

    2. Start the zookeeper client

      zookeeper-client -server $(hostname -f):2181
    3. Enter the following to reset the ACLs of the root node: 
      setAcl / world:anyone:crdwa

Once Kafka metadata in Zookeeper is restricted via ACLS, administrative operations, for example topic creation, deletion, any configuration changes and so on, can only be performed by authorized users.

Unlocking Kafka Metadata in Zookeeper

Prerequisites

A secure Kafka cluster with Kerberos authentication enabled is required. For more information see, Enabling Kerberos Authentication.

Steps

In order to unrestrict access to Kafka metadata stored in Zookeeper by completing the following steps:
  1. Disable the use of secure ACLs by setting zookeeper.set.acl configuration parameter to false.
    1. In Cloudera Manager go to the Kafka service.
    2. Select Configuration and find the Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka.properties property.
    3. Add the following line to the Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka.properties property: 
      zookeeper.set.acl=false
  2. Perform a Rolling Restart
    1. Return to the Home page by clicking the Cloudera Manager logo.
    2. Go to the Kafka service and select Actions Rolling Restart.
    3. Check the Restart roles with stale configurations only checkbox and click Rolling restart.
    4. Click Close when the restart has finished.
  3. Run the zookeeper-security-migration tool with the zookeeper.acl option set to unsecure. 
    zookeeper-security-migration --zookeeper.connect hostname:port --zookeeper.acl unsecure

    The tool traverses the corresponding sub-trees changing the ACLs of the znodes.