Cloudera Docs

Hardening ZooKeeper Znodes for NiFi security

By default, the ZooKeeper Znodes used by NiFi are not secured, which can expose sensitive data to unauthorized access. This section provides instructions for identifying, validating, and hardening the ZooKeeper Znodes used by NiFi. These steps are essential for securing NiFi’s interaction with ZooKeeper in a production environment.

Preparations

Run the following commands on one of your NiFi nodes.

  1. Identify the unhardened Znode.

    By default, the Znode used by NiFi in Zookeeper is not secured. To identify the root Znode used by NiFi in your Zookeeper cluster, follow these steps:

    1. Identify the Znode used by NiFi. 
      grep "nifi.zookeeper.root.node" $(ps -aux | grep -Eo "[***/var/[^=]*nifi\.properties***]")

      This command checks the nifi.properties file for the nifi.zookeeper.root.node property and outputs the exact Znode name. For example:

      [root@ccycloud-4.quasar-nncuew.root.comops.site 215-nifi-NIFI_NODE]# grep "nifi.zookeeper.root.node" nifi.properties
nifi.zookeeper.root.node=/1

      Use this Znode name in all subsequent steps to ensure you are modifying the correct configuration.

    2. Verify the current Znode configuration.

      Once you have determined your Znode, you can validate it and its subnodes in Zookeeper, using the following commands. For demonstration, in the syntax Znode is referred to as /BASE. Replace it with your determined Znode name.

      1. List the contents of the Znode:
        [zk: localhost:2181(CONNECTED) 2] ls /BASE
[leaders]
      2. Check the access control list (ACL) of the Znode:
        [zk: localhost:2181(CONNECTED) 3] getAcl /BASE
'world,'anyone
: cdrwa
      3. Check the ACL of subnodes:
        [zk: localhost:2181(CONNECTED) 4] getAcl /BASE/leaders
'world,'anyone
: cdrwa
  2. Prepare the JAAS file.
    1. Create a JAAS configuration file to Kerberize the NiFi ZooKeeper client.
      Example configuration:
      Client {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  keyTab="/hadoopfs/fs1/working-dir/nifi.keytab"
  storeKey=true
  useTicketCache=false
principal="nifi/zz-mwies-nifi-nifi1.zz-mwies.a465-9q4k.cloudera.site@ZZ-MWIES.A465-9Q4K.CLOUDERA.SITE";
};
      Keytab path
      The keytab file path is typically located where nifi.working.directory is configured. For on-premises deployments, the default path is /var/lib/nifi, unless otherwise specified.
      Per-node configuration
      If you have multiple nodes, remember that the principal name is unique to each node because it includes the hostname. You will need to create a slightly modified JAAS configuration for each node to reflect its specific principal.
      Principal
      To determine the exact principal:
      1. Log in to the Cloudera Manager UI.
      2. Go to Administration > Security > Kerberos Credentials.
      3. Locate the principal for NiFi.
    2. Save this file in a location accessible to NiFi, such as /etc/nifi-jaas/jaas.conf.
  3. Validate the JAAS file.

    To ensure the JAAS configuration file is valid, test it against Zookeeper to see if SASL authentication is successful.

    1. Export the JVM flags. 
      export CLIENT_JVMFLAGS="-Djava.security.auth.login.config=/etc/nifi-jaas/jaas.conf"
    2. Connect to ZooKeeper. 
      zookeeper-client -server $(hostname -f):2181
    3. Confirm the output. 
      Connecting to zz-mwies-nifi-nifi1.zz-mwies.a465-9q4k.cloudera.site:2181
Welcome to ZooKeeper!
JLine support is enabled
[zk: zz-mwies-nifi-nifi1.zz-mwies.a465-9q4k.cloudera.site:2181(CONNECTING) 0] 
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
WATCHER::
WatchedEvent state:SaslAuthenticated type:None path:null
    4. Optional: If you see state:SaslAuthenticated, the JAAS file has been created correctly. If there are any issues, use the cat -vet command to identify and fix malformations in the file.
    5. Distribute the JAAS files to each NiFi node and change the hostname accordingly.

Reconfiguring NiFi

This section guides you through reconfiguring NiFi for secure communication with ZooKeeper using Kerberos authentication. The process involves updating several NiFi configuration files with advanced settings (safety valves) to ensure proper integration and security.

  1. For each safety valve, add the corresponding XML snippet.

    NiFi Node Advanced Configuration Snippet (Safety Valve) for staging/bootstrap.conf.xml:

    <property><name>java.arg.15</name><value>-Djava.security.auth.login.config=/etc/nifi-jaas/jaas.conf</value></property>

    NiFi Node Advanced Configuration Snippet (Safety Valve) for staging/nifi.properties.xml:

    <property><name>nifi.zookeeper.auth.type</name><value>sasl</value></property><property><name>nifi.zookeeper.kerberos.removeHostFromPrincipal</name><value>true</value></property><property><name>nifi.zookeeper.kerberos.removeRealmFromPrincipal</name><value>true</value></property>

    NiFi Node Advanced Configuration Snippet (Safety Valve) for staging/state-management.xml:

    <property><name>xml.state-management.cluster-provider.zk-provider.property.Access Control</name><value>CreatorOnly</value><final>true</final></property>
  2. Save all the changes made to the configuration files.
  3. Stop NiFi.

Znode hardening

  1. Back up Components Znode.

    Before proceeding, in case there is a components Znode, ensure a backup of it is created.

    /opt/cloudera/parcels/CFM-***[version]***/TOOLKIT/bin/zk-migrator.sh -r -z $(hostname -f):2181/BASE/components -k /etc/nifi-jaas/jaas.conf -f components-backup.json

    This command saves the content of components Znode into a JSON file.

  2. Recreate Components Znode.
    1. Delete the existing components Znode in Zookeeper. 
      [zk: zz-mwies-nifi-nifi2.zz-mwies.a465-9q4k.cloudera.site:2181(CONNECTED) 0] deleteall /BASE/components
[zk: zz-mwies-nifi-nifi2.zz-mwies.a465-9q4k.cloudera.site:2181(CONNECTED) 1] ls /BASE
[]
[zk: zz-mwies-nifi-nifi2.zz-mwies.a465-9q4k.cloudera.site:2181(CONNECTED) 2] quit
WATCHER::
WatchedEvent state:Closed type:None path:null
    2. Re-import the components Znode from the backup file. 
      [root@zz-mwies-nifi-nifi2 backup]# /opt/cloudera/parcels/CFM-***[version]***/TOOLKIT/bin/zk-migrator.sh -s -z $(hostname -f):2181/BASE/components -k /etc/nifi-jaas/jaas.conf -f components-backup.json --ignore-source
    3. After recreating the components Znode, verify that it has the correct permissions and content. 
      [zk: zz-mwies-nifi-nifi2.zz-mwies.a465-9q4k.cloudera.site:2181(CONNECTED) 0] ls /BASE
[components]
[zk: zz-mwies-nifi-nifi2.zz-mwies.a465-9q4k.cloudera.site:2181(CONNECTED) 1] getAcl /BASE/components
'sasl,'nifi
: cdrwa
[zk: zz-mwies-nifi-nifi2.zz-mwies.a465-9q4k.cloudera.site:2181(CONNECTED) 2] ls -R /BASE/components
/BASE/components
/BASE/components/0c2f3aaf-0188-1000-ffff-ffffa939f0e7
[zk: zz-mwies-nifi-nifi2.zz-mwies.a465-9q4k.cloudera.site:2181(CONNECTED) 3] getAcl /BASE/components/0c2f3aaf-0188-1000-ffff-ffffa939f0e7
'sasl,'nifi
: cdrwa
[zk: zz-mwies-nifi-nifi2.zz-mwies.a465-9q4k.cloudera.site:2181(CONNECTED) 4]
  3. Set the permissions for BASE Znode.

    To harden the BASE Znode, open Zookeeper and set the appropriate ACL using the following command.

    [zk: zz-mwies-nifi-nifi2.zz-mwies.a465-9q4k.cloudera.site:2181(CONNECTED) 5] setAcl /BASE sasl:nifi:cdrwa,world:anyone:r
[zk: zz-mwies-nifi-nifi2.zz-mwies.a465-9q4k.cloudera.site:2181(CONNECTED) 6] getAcl /BASE
'sasl,'nifi
: cdrwa
'world,'anyone
: r

    If you have a /BASE/leaders Znode, set its ACL recursively.

    Similarly, set the ACL for /BASE/components with the same permissions:

    setAcl /BASE/components sasl:nifi:cdrwa,world:anyone:r
  4. Start NiFi.

    Once all configurations are verified and updated, you can safely start the NiFi service.