Configuring TLS/SSL for Flume

Several Flume components support the TLS/SSL protocols to communicate with other systems securely.

Flume Components Supporting TLS/SSL
Component SSL server or client
Avro Source server
Avro Sink client
Thrift Source server
Thrift Sink client
Kafka Source client
Kafka Channel client
Kafka Sink client
HTTP Source server
JMS Source client
Syslog TCP Source server
Multiport Syslog TCP Source server

Minimum Required Role: Configurator (also provided by Cluster Administrator, Full Administrator)

Setting Keystore and Truststore for Flume Components

You can set up keystore and truststore for Flume at the component level or globally. In case of global setup, you can define the keystore / truststore properties once in Cloudera Manager, and use the same settings for all components.

You can find the Keystore File Location, Keystore File Password, Trust Store File, and Trust Store Password properties on the Flume configuration page in Cloudera Manager.



The keystore and truststore properties defined in Cloudera Manager provide the 'default' keystore and truststore for Flume components that have SSL enabled. For component level setup, you can configure keystore and truststore in the agent configuration file, through component specific parameters.

Auto-TLS

Auto-TLS support is available in Cloudera Manager, both keystore and truststore properties support Auto-TLS.



Enabling TLS/SSL for Flume Components

The usual Enable TLS/SSL property is not available for Flume in Cloudera Manager. Enabling TLS/SSL for individual Flume components is always done at component level, in the agent configuration file.

To enable SSL for SSL-capable sources (except JMS and Kafka source), use: 
a1.sources.r1.ssl=true
To enable SSL for SSL-capable sinks (except Kafka sink), use: 
a1.sinks.s1.ssl=true
Enabling SSL for JMS source and Kafka components is handled differently. You can find details on their TLS/SSL configuration at:

Configuring TLS/SSL for Flume JMS Source

There are no component-level configuration properties and no Enable TLS/SSL flag for Flume's JMS sources unlike for other components. SSL setup is controlled by JNDI/Provider URL settings (ultimately the JMS server settings), and by the presence / absence of truststore and keystore.

JMS client implementations usually support configuring TLS/SSL via Java system properties defined by Java Secure Socket Extension (JSSE). You can configure these properties in Cloudera Manager. If you specify these system properties for Flume's JVM, its JMS source can connect to the JMS server through SSL.

The following sections provide information on SSL configuration on the Flume side only, assuming that the JMS server has already been configured for SSL correctly.

Configuring SSL Transport / Server Authentication

If the JMS server uses a self-signed certificate or its certificate is signed by a non-trusted certificate authority (CA), for example, the company's own CA, you have to set up a truststore that contains the right certificate, and pass it to Flume.

To set up the truststore, edit the Flume TLS/SSL Client Trust Store File and Flume TLS/SSL Client Trust Store Password properties in Cloudera Manager.

  1. Open the Cloudera Manager Admin Console.
  2. Go to the Flume service.
  3. Click the Configuration tab.
  4. Select Scope > Flume.
  5. Select Category > Security. The TLS/SSL configuration options display.
  6. Edit the Flume TLS/SSL Client Trust Store File and Flume TLS/SSL Client Trust Store Password properties.



  7. Enter a Reason for change, and then click Save Changes to commit the changes.
  8. Restart the Flume service.
If you want to use IBM MQ with Flume as a JMS source, you must provide an additional Java system property because of the mismatch between the cipher suite names used in IBM and Oracle JVMs. You can configure this Java system property in the Java Configuration Options for Flume Agent field in Cloudera Manager: 
-Dcom.ibm.mq.cfg.useIBMCipherMappings=false
If you want to turn on debug logging for SSL, specify the following Java system property in the Java Configuration Options for Flume Agent field in Cloudera Manager: 
-Djavax.net.debug=ssl

Some JMS providers require SSL-specific JNDI Initial Context Factory and/or Provider URL settings when using SSL. For example, ActiveMQ uses ssl:// URL prefix instead of tcp://. In this case, you must adjust the JMS source properties (initialContextFactory and/or providerURL) in the Flume agent configuration file (Configuration File field) in Cloudera Manager.

Configuring Client Certificate Authentication (Two-Way SSL)

The JMS source can authenticate to the JMS server through client certificate authentication (instead of the usual user/password login) when SSL is used, and the JMS server is configured to accept this kind of authentication.

To configure the keystore that contains Flume's authentication key, fill in the Flume TLS/SSL Server JKS Keystore File Location and Flume TLS/SSL Server JKS Keystore File Password properties.

  1. Open the Cloudera Manager Admin Console.
  2. Go to the Flume service.
  3. Click the Configuration tab.
  4. Select Scope > Flume.
  5. Select Category > Security. The TLS/SSL configuration options display.
  6. Edit the Flume TLS/SSL Server Keystore File and Flume TLS/SSL Server Keystore Password properties.



  7. Enter a Reason for change, and then click Save Changes to commit the changes.
  8. Restart the Flume service.

Configuring TLS/SSL for Flume Thrift Source and Sink

You can configure TLS/SSL for each Thrift source and sink instance either in Cloudera Manager or use the command line to edit the flume.conf file.

The following tables list the properties that must be configured to enable TLS/SSL communication between Flume's Thrift source and sink instances.

Thrift Source TLS/SSL Properties
Property Description
ssl Set to true to enable TLS/SSL encryption.
keystore (Optional) The path to a Java keystore file. Keystore parameters can also be defined in the global keystore fields in CM.
keystore-password (Optional) The password for the Java keystore. Keystore parameters can also be defined in the global keystore fields in CM.
keystore-type The type of the Java keystore. This can be JKS or PKCS12.
Thrift Sink TLS/SSL Properties
Property Description
ssl Set to true to enable TLS/SSL for this ThriftSink.

When configuring TLS/SSL, you can optionally set the following truststore, truststore-password and truststore-type properties. If a custom truststore is not specified, Flume will use the default Java JSSE truststore (typically jssecacerts or cacerts in the Oracle JRE) to verify the remote Thrift Source's TLS/SSL credentials.
truststore (Optional) The path to a custom Java truststore file.
truststore-password (Optional) The password for the specified truststore.
truststore-type (Optional) The type of the Java truststore. This can be JKS or any other supported Java truststore type.

Configuring TLS/SSL in the flume.conf File

Minimum Required Role: Configurator (also provided by Cluster Administrator, Full Administrator)

For agent a1, source r1, and sink k1, you would add the following properties: 
# TLS/SSL properties for Thrift source r1 
a1.sources.r1.ssl=true
a1.sources.r1.keystore=<path/to/keystore>
a1.sources.r1.keystore-password=<keystore password>
a1.sources.r1.keystore-type=<keystore type>

# TLS/SSL properties for Thrift sink k1 
a1.sinks.k1.ssl=true
a1.sinks.k1.truststore=<path/to/truststore>
a1.sinks.k1.truststore-password=<truststore password>
a1.sinks.k1.truststore-type=<truststore type>

Configuring TLS/SSL Using Cloudera Manager

Minimum Required Role: Configurator (also provided by Cluster Administrator, Full Administrator)

  1. Open the Cloudera Manager Admin Console.
  2. Go to the Flume Service.
  3. Click the Configuration tab.
  4. Select Scope > Agent.
  5. Select Category > Main.
  6. Edit the Configuration File property and add the Thrift source and sink properties for each Thrift source and sink instance as described in Configuring TLS/SSL in the flume.conf File.
  7. Enter a Reason for change, and then click Save Changes to commit the changes.
  8. Restart the Flume service.