Step 10: (Flume Only) Use Substitution Variables for the Kerberos Principal and Keytab

Minimum Required Role: Configurator (also provided by Cluster Administrator, Full Administrator)

As described in Flume security configuration, if you are using Flume on a secure cluster you must configure the HDFS sink with the following configuration options in the flume.conf file:

  • hdfs.kerberosPrincipal - fully-qualified principal.
  • hdfs.kerberosKeytab - location on the local host of the keytab containing the user and host keys for the above principal

Here is an example of an HDFS sink configuration in the flume.conf file (the majority of the HDFS sink configuration options have been omitted):

agent.sinks.sink-1.type = HDFS
agent.sinks.sink-1.hdfs.kerberosPrincipal = flume/_HOST@YOUR-REALM.COM
agent.sinks.sink-1.hdfs.kerberosKeytab = /etc/flume-ng/conf/flume.keytab
agent.sinks.sink-1.hdfs.proxyUser = weblogs

Since Cloudera Manager generates the Flume keytab files for you, and the locations of the keytab files cannot be known beforehand, substitution variables are required for Flume. Cloudera Manager provides two Flume substitution variables called $KERBEROS_PRINCIPAL and $KERBEROS_KEYTAB to configure the principal name and the keytab file path respectively on each host.

Here is an example of using the substitution variables to configure the options shown in the previous example:

agent.sinks.sink-1.type = hdfs
agent.sinks.sink-1.hdfs.kerberosPrincipal = $KERBEROS_PRINCIPAL
agent.sinks.sink-1.hdfs.kerberosKeytab = $KERBEROS_KEYTAB
agent.sinks.sink-1.hdfs.proxyUser = weblogs

Use the following instructions to have Cloudera Manager add these variables to the flume.conf file on every host that Cloudera Manager manages.

To use the Flume substitution variables for the Kerberos principal and keytab:

  1. Go to the Flume service > Configuration page in Cloudera Manager.
  2. Click Agent.
  3. In the Configuration File property, add the configuration options with the substitution variables. For example: 
    agent.sinks.sink-1.type = hdfs
agent.sinks.sink-1.hdfs.kerberosPrincipal = $KERBEROS_PRINCIPAL
agent.sinks.sink-1.hdfs.kerberosKeytab = $KERBEROS_KEYTAB
agent.sinks.sink-1.hdfs.proxyUser = weblogs
  4. Click Save.