Step 10: (Flume Only) Use Substitution Variables for the Kerberos Principal and Keytab

Minimum Required Role: Configurator (also provided by Cluster Administrator, Full Administrator)

As described in Flume security configuration, if you are using Flume on a secure cluster you must configure the HDFS sink or HBase sink with the following configuration options in the flume.conf file:

  • kerberosPrincipal - fully qualified principal.
  • kerberosKeytab - location on the local host of the keytab containing the user and host keys for the above principal

Since Cloudera Manager generates the Flume keytab files for you, and the locations of the keytab files cannot be known beforehand, substitution variables are required for Flume. Cloudera Manager provides two Flume substitution variables called $KERBEROS_PRINCIPAL and $KERBEROS_KEYTAB to configure the principal name and the keytab file path respectively on each host.

HDFS Sink Example

The following example shows an HDFS sink configuration in the flume.conf file (the majority of the HDFS sink configuration options have been omitted):

agent.sinks.sink-1.type = HDFS
agent.sinks.sink-1.hdfs.kerberosPrincipal = flume/_HOST@YOUR-REALM.COM
agent.sinks.sink-1.hdfs.kerberosKeytab = /etc/flume-ng/conf/flume.keytab
agent.sinks.sink-1.hdfs.proxyUser = weblogs

The text below shows the same configuration options with the substitution variables:

agent.sinks.sink-1.type = hdfs
agent.sinks.sink-1.hdfs.kerberosPrincipal = $KERBEROS_PRINCIPAL
agent.sinks.sink-1.hdfs.kerberosKeytab = $KERBEROS_KEYTAB
agent.sinks.sink-1.hdfs.proxyUser = weblogs

HBase Sink Example

The following example shows an HBase sink configuration in the flume.conf file (the majority of the HBase sink configuration options have been omitted):

agent.sinks.sink-1.type = hbase
agent.sinks.sink-1.kerberosPrincipal = flume/_HOST@YOUR-REALM.COM
agent.sinks.sink-1.kerberosKeytab = /etc/flume-ng/conf/flume.keytab

The text below shows the same configuration options with the substitution variables:

agent.sinks.sink-1.type = hbase
agent.sinks.sink-1.kerberosPrincipal = $KERBEROS_PRINCIPAL
agent.sinks.sink-1.kerberosKeytab = $KERBEROS_KEYTAB

Use the Flume Substitution Variables for the Kerberos Principal and Keytab

Complete the following steps to have Cloudera Manager add these variables to the flume.conf file on every host that Cloudera Manager manages.

  1. Go to the Flume service > Configuration page in Cloudera Manager.
  2. Click Agent.
  3. In the Configuration File property, add the configuration options with the substitution variables. For example: 
    agent.sinks.sink-1.type = hdfs
agent.sinks.sink-1.hdfs.kerberosPrincipal = $KERBEROS_PRINCIPAL
agent.sinks.sink-1.hdfs.kerberosKeytab = $KERBEROS_KEYTAB
agent.sinks.sink-1.hdfs.proxyUser = weblogs
  4. Click Save.