Encrypted Shuffle and Encrypted Web UIs

Once you've enabled Kerberos, which provides for strong authentication, you can optionally enable network encryption if you so desire. CDH 5 supports the Encrypted Shuffle and Encrypted Web UIs feature that allows encryption of the MapReduce shuffle and web server ports using HTTPS with optional client authentication (also known as bi-directional HTTPS, or HTTPS with client certificates). It includes:

Hadoop configuration setting for toggling the shuffle between HTTP and HTTPS.
Hadoop configuration setting for toggling the Web UIs to use either HTTP or HTTPS.
Hadoop configuration settings for specifying the keystore and truststore properties (location, type, passwords) that are used by the shuffle service, web server UIs and the reducers tasks that fetch shuffle data.
A way to re-read truststores across the cluster (when a node is added or removed).

CDH 5 supports Encrypted Shuffle for both MRv1 and MRv2 (YARN), with common configuration properties used for both versions. The only configuration difference is in the parameters used to enable the features:

For MRv1, setting the hadoop.ssl.enabled parameter in the core-site.xml file enables both the Encrypted Shuffle and the Encrypted Web UIs. In other words, the encryption toggling is coupled for the two features.
For MRv2, setting the hadoop.ssl.enabled parameter enables the Encrypted Web UI feature; setting the mapreduce.shuffle.ssl.enabled parameter in the mapred-site.xml file enables the Encrypted Shuffle feature.

All other configuration properties apply to both the Encrypted Shuffle and Encrypted Web UI functionality.

When the Encrypted Web UI feature is enabled, all Web UIs for Hadoop components are served over HTTPS. If you configure the systems to require client certificates, browsers must be configured with the appropriate client certificates in order to access the Web UIs.

Most components that run on top of MapReduce automatically use Encrypted Shuffle when it is configured.

Continue reading:

Configuring Encrypted Shuffle and Encrypted Web UIs
Activating Encrypted Shuffle
Client Certificates
Reloading Truststores
Debugging

Configuring Encrypted Shuffle and Encrypted Web UIs

To configure Encrypted Shuffle and Encrypted Web UIs, set the appropriate property/value pairs in the following:

core-site.xml enables these features and defines the implementation
mapred-site.xml enables Encrypted Shuffle for MRv2
ssl-server.xml stores keystone and truststore settings for the server
ssl-client.xml stores keystone and truststore settings for the client

core-site.xml Properties

To configure encrypted shuffle, set the following properties in the core-site.xml files of all nodes in the cluster:

Property	Default Value	Explanation
`hadoop.ssl.enabled`	`false`	For MRv1, setting this value to `true` enables both the Encrypted Shuffle and the Encrypted Web UI features. For MRv2, this property only enables the Encrypted WebUI; Encrypted Shuffle is enabled with a property in the `mapred-site.xml` file as described below.
`hadoop.ssl.require.client.cert`	`false`	When this property is set to `true`, client certificates are required for all shuffle operations and all browsers used to access Web UIs. Cloudera recommends that this be set to `false`. See Client Certificates.

Property

Default Value

Explanation

hadoop.ssl.enabled

false

For MRv1, setting this value to true enables both the Encrypted Shuffle and the Encrypted Web UI features. For MRv2, this property only enables the Encrypted WebUI; Encrypted Shuffle is enabled with a property in the mapred-site.xml file as described below.

hadoop.ssl.require.client.cert

false

When this property is set to true, client certificates are required for all shuffle operations and all browsers used to access Web UIs.

Cloudera recommends that this be set to false. See Client Certificates.

`hadoop.ssl.hostname.verifier`	`DEFAULT`	The hostname verifier to provide for HttpsURLConnections. Valid values are: `DEFAULT`, `STRICT`, `STRICT_I6`, `DEFAULT_AND_LOCALHOST` and `ALLOW_ALL`.
`hadoop.ssl.keystores.factory.class`	`org.apache.hadoop` `.security.ssl`. `FileBasedKeyStoresFactory`	The `KeyStoresFactory` implementation to use.
`hadoop.ssl.server.conf`	`ssl-server.xml`	Resource file from which ssl server keystore information is extracted. This file is looked up in the `classpath`; typically it should be in the `/etc/hadoop/conf/` directory.
`hadoop.ssl.client.conf`	`ssl-client.xml`	Resource file from which ssl server keystore information is extracted. This file is looked up in the `classpath`; typically it should be in the `/etc/hadoop/conf/` directory.

Example

<configuration>
    ...
    <property>
      <name>hadoop.ssl.require.client.cert</name>
      <value>false</value>
      <final>true</final>
    </property>

    <property>
      <name>hadoop.ssl.hostname.verifier</name>
      <value>DEFAULT</value>
      <final>true</final>
    </property>

    <property>
      <name>hadoop.ssl.keystores.factory.class</name>
      <value>org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory</value>
      <final>true</final>
    </property>

    <property>
      <name>hadoop.ssl.server.conf</name>
      <value>ssl-server.xml</value>
      <final>true</final>
    </property>

    <property>
      <name>hadoop.ssl.client.conf</name>
      <value>ssl-client.xml</value>
      <final>true</final>
    </property>

    <property>
      <name>hadoop.ssl.enabled</name>
      <value>true</value>
    </property>
    ...
</configuration>

The cluster should be configured to use the Linux Task Controller in MRv1 and Linux container executor in MRv2 to run job tasks so that they are prevented from reading the server keystore information and gaining access to the shuffle server certificates. Refer to Information about Other Hadoop Security Programs for more information.

mapred-site.xml Property (MRv2 only)

To enable Encrypted Shuffle for MRv2, set the following property in the mapred-site.xml file on every node in the cluster:

Property	Default Value	Explanation
`mapreduce.shuffle.ssl.enabled`	`false`	If this property is set to `true`, encrypted shuffle is enabled. If this property is not specified, it defaults to the value of `hadoop.ssl.enabled`. This value can be `false` when `hadoop.ssl.enabled` is `true` but cannot be `true` when `hadoop.ssl.enabled` is `false`

This property should be marked as final in the cluster configuration files.

Example:

<configuration>
    ...
    <property>
      <name>mapreduce.shuffle.ssl.enabled</name>
      <value>true</value>
      <final>true</final>
    </property>
    ...
</configuration>

Keystore and Truststore Settings

FileBasedKeyStoresFactory is the only KeyStoresFactory that is currently implemented. It uses properties in the ssl-server.xml and ssl-client.xml files to configure the keystores and truststores.

ssl-server.xml (Shuffle server and Web UI) Configuration

Use the following settings to configure the keystores and truststores in the ssl-server.xml file.

Property	Default Value	Description
`ssl.server.keystore.type`	`jks`	Keystore file type
`ssl.server.keystore.location`	NONE	Keystore file location. The `mapred` user should own this file and have exclusive read access to it.
`ssl.server.keystore.password`	NONE	Keystore file password
`ssl.server.keystore.keypassword`	NONE	Key password
`ssl.server.truststore.type`	`jks`	Truststore file type
`ssl.server.truststore.location`	NONE	Truststore file location. The `mapred` user should own this file and have exclusive read access to it.
`ssl.server.truststore.password`	NONE	Truststore file password
`ssl.server.truststore.reload.interval`	10000	Truststore reload interval, in milliseconds

Example

<configuration>
<!-- Server Certificate Store -->
<property>
  <name>ssl.server.keystore.type</name>
     <value>jks</value>
</property>
<property>
  <name>ssl.server.keystore.location</name>
  <value>${user.home}/keystores/server-keystore.jks</value>
</property>
<property>
  <name>ssl.server.keystore.password</name>
  <value>serverfoo</value>
</property>
<property>
  <name>ssl.server.keystore.keypassword</name>
  <value>serverfoo</value>
</property>

<!-- Server Trust Store -->
<property>
  <name>ssl.server.truststore.type</name>
  <value>jks</value>
</property>
<property>
  <name>ssl.server.truststore.location</name>
  <value>${user.home}/keystores/truststore.jks</value>
</property>
<property>
  <name>ssl.server.truststore.password</name>
  <value>clientserverbar</value>
</property>
<property>
  <name>ssl.server.truststore.reload.interval</name>
  <value>10000</value>
</property>
</configuration>

ssl-client.xml (Reducer/Fetcher) Configuration

Use the following settings to configure the keystores and truststores in the ssl-client.xml file. This file should be owned by the mapred user for MRv1 and by the yarn user for MRv2; the file permissions should be 444 (read access for all users).

Property	Default Value	Description
`ssl.client.keystore.type`	`jks`	Keystore file type
`ssl.client.keystore.location`	NONE	Keystore file location. The mapred user should own this file and it should have default permissions.
`ssl.client.keystore.password`	NONE	Keystore file password
`ssl.client.keystore.keypassword`	NONE	Key password
`ssl.client.truststore.type`	`jks`	Truststore file type
`ssl.client.truststore.location`	NONE	Truststore file location. The mapred user should own this file and it should have default permissions.
`ssl.client.truststore.password`	NONE	Truststore file password
`ssl.client.truststore.reload.interval`	10000	Truststore reload interval, in milliseconds

Example

<configuration>

  <!-- Client certificate Store -->
  <property>
    <name>ssl.client.keystore.type</name>
    <value>jks</value>
  </property>
  <property>
    <name>ssl.client.keystore.location</name>
    <value>${user.home}/keystores/client-keystore.jks</value>
  </property>
  <property>
    <name>ssl.client.keystore.password</name>
    <value>clientfoo</value>
  </property>
  <property>
    <name>ssl.client.keystore.keypassword</name>
    <value>clientfoo</value>
  </property>

  <!-- Client Trust Store -->
  <property>
    <name>ssl.client.truststore.type</name>
    <value>jks</value>
  </property>
  <property>
    <name>ssl.client.truststore.location</name>
    <value>${user.home}/keystores/truststore.jks</value>
  </property>
  <property>
    <name>ssl.client.truststore.password</name>
    <value>clientserverbar</value>
  </property>
  <property>
    <name>ssl.client.truststore.reload.interval</name>
    <value>10000</value>
  </property>
</configuration>

Activating Encrypted Shuffle

When you have made the above configuration changes, activate Encrypted Shuffle by re-starting all TaskTrackers in MRv1 and all NodeManagers in YARN.

Client Certificates

Client Certificates are supported but they do not guarantee that the client is a reducer task for the job. The Client Certificate keystore file that contains the private key must be readable by all users who submit jobs to the cluster, which means that a rogue job could read those keystore files and use the client certificates in them to establish a secure connection with a Shuffle server. The JobToken mechanism that the Hadoop environment provides is a better protector of the data; each job uses its own JobToken to retrieve only the shuffle data that belongs to it. Unless the rogue job has a proper JobToken, it cannot retrieve Shuffle data from the Shuffle server.

Reloading Truststores

By default, each truststore reloads its configuration every 10 seconds. If a new truststore file is copied over the old one, it is re-read, and its certificates replace the old ones. This mechanism is useful for adding or removing nodes from the cluster, or for adding or removing trusted clients. In these cases, the client, TaskTracker or NodeManager certificate is added to (or removed from) all the truststore files in the system, and the new configuration is picked up without requiring that the TaskTracker in MRv1 and NodeManager in YARN daemons are restarted.

The reload interval is controlled by the ssl.client.truststore.reload.interval and ssl.server.truststore.reload.interval configuration properties in the ssl-client.xml and ssl-server.xml files described above.

Debugging

To enable TLS/SSL debugging in the reducers, set -Djavax.net.debug=all in the mapred.reduce.child.java.opts property; for example:

<configuration>
    ...
    <property>
        <name>mapred.reduce.child.java.opts</name>
        <value>-Xmx200m -Djavax.net.debug=all</value>
    </property>
    ...
</configuration>

You can do this on a per-job basis, or by means of a cluster-wide setting in mapred-site.xml.

To set this property in TaskTrackersfor MRv1, set it in hadoop-env.sh:

HADOOP_TASKTRACKER_OPTS="-Djavax.net.debug=all $HADOOP_TASKTRACKER_OPTS"

To set this property in NodeManagers for YARN, set it in hadoop-env.sh:

YARN_OPTS="-Djavax.net.debug=all $YARN_OPTS"

Categories: Configuring | Encryption | Security | All Categories

Configuring TLS/SSL for HttpFS

Deployment Planning for Data at Rest Encryption