ZooKeeper ACLs Best Practices: HDFS/WebHDFS

You must follow the best practices for tightening the ZooKeeper ACLs or permissions for HDFS/WebHDFS when provisioning a secure cluster.

ZooKeeper Usage:
- hadoop-ha - hdfs zkfc automatic NameNode failover
Default ACLs:
- hadoop-ha - world: anyone:cdrwa
Security Best Practice ACLs/Permissions and Required Steps:
- hadoop-ha - sasl: nn:cdrwa
Existing SmartSense rule recommends ACL of sasl:nn:rwcda for secured clusters. To set this:
1. Set ha.zookeeper.acl to sasl:nn:rwcda:
  - Using Ambari:
    
    Add ha.zookeeper.acl with value sasl:nn:rwcda in Configs>Advanced>Custom core-site.
  - Manually:
    
    Add this to core-site. xml as root user:
    <property> <name>ha.zookeeper.acl</name> <value>sasl:nn:rwcda</value> </property>
2. Add this HADOOP_ZKFC_OPTS export:
  - Using Ambari:
    In Configs > Advanced > Advanced hadoop-env > hadoop-env template, add the following:
    export HADOOP_ZKFC_OPTS="Dzookeeper.sasl.client=true Dzookeeper.sasl.client.username=zookeeper Djava.security.auth.login.config=/etc/hadoop/conf/hdfs_jaas.conf Dzookeeper.sasl.clientconfig=Client ${HADOOP_ZKFC_OPTS}"
  - Manually:
    Add this to hadoop-env.sh as root user:
    export HADOOP_ZKFC_OPTS="Dzookeeper.sasl.client=true Dzookeeper.sasl.client.username=zookeeper Djava.security.auth.login.config=/etc/hadoop/conf/hdfs_jaas.conf Dzookeeper.sasl.clientconfig=Client ${HADOOP_ZKFC_OPTS}"
3. On two Namenodes, create /etc/hadoop/conf/hdfs_jaas.conf as root user with the following contents:
```
Client {
          com.sun.security.auth.module.Krb5LoginModule required
          useKeyTab=true
          storeKey=true
          useTicketCache=false
          keyTab="/etc/security/keytabs/nn.service.keytab"
          principal="nn/<HOST>@EXAMPLE.COM";
};
```
  nn/<HOST>@EXAMPLE.COM must be changed to the actual hostname and realm, e.g. nn/c6401.ambari.apache.org@EXAMPLE.COM. To get actual principal, on two Namenodes, run the command as hdfs user: klist -k /etc/security/keytabs/nn.service.keytab.
4. Stop the two ZKFCs.
5. On one of Namenodes, run the command as hdfs user: hdfs zkfc -formatZK -force.
6. Start the two ZKFCs.
One of two Namenodes may be stopped in the process, or standby Namenode may be transitioned to active one. Start the stopped namenode if any.