Security
Also available as:
PDF
loading table of contents...

Enable SSL for WebHDFS, MapReduce Shuffle, Tez, and YARN

This section explains how to set up SSL for WebHDFS, YARN and MapReduce. Before you begin, make sure that the SSL certificate is properly configured, including the keystore and truststore that will be used by WebHDFS, MapReduce, and YARN.

HDP supports the following SSL modes:

  • One-way SSL: SSL client validates the server identity only.

  • Mutual authentication (2WAY SSL): The server and clients validate each others' identities. 2WAY SSL can cause performance delays and is difficult to set up and maintain.

[Note]Note

In order to access SSL enabled HDP Services through the Knox Gateway, additional configuration on the Knox Gateway is required, see Configure SSL for Knox.

To enable one-way SSL set the following properties and restart all services:

  1. Set the following property values (or add the properties if required) in core-site.xml:

    hadoop.ssl.require.client.cert=false

    hadoop.ssl.hostname.verifier=DEFAULT

    hadoop.ssl.keystores.factory.class=org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory

    hadoop.ssl.server.conf=ssl-server.xml

    hadoop.ssl.client.conf=ssl-client.xml

    [Note]Note

    Specify the hadoop.ssl.server.conf and hadoop.ssl.client.conf values as the relative or absolute path to Hadoop SSL Keystore Factory configuration files. If you specify only the file name, put the files in the same directory as the core-site.xml.

  2. Set the following properties (or add the properties if required) in hdfs-site.xml:

    • dfs.http.policy=<Policy>

    • dfs.client.https.need-auth=true (optional for mutual client/server certificate validation)

    • dfs.datanode.https.address=<hostname>:50475

    • dfs.namenode.https-address=<hostname>:50470

      where <Policy> is either:

      • HTTP_ONLY: service is provided only on HTTP

      • HTTPS_ONLY: service is provided only on HTTPS

      • HTTP_AND_HTTPS: service is provided both on HTTP and HTTPS

  3. Set the following properties in mapred-site.xml:

    mapreduce.jobhistory.http.policy=HTTPS_ONLY
    mapreduce.jobhistory.webapp.https.address=<JHS>:<JHS_HTTPS_PORT> 
    mapreduce.ssl.enabled=true
    mapreduce.shuffle.ssl.enabled=true 
  4. Set the following properties in yarn-site.xml:

    yarn.http.policy=HTTPS_ONLY
    yarn.log.server.url=https://<JHS>:<JHS_HTTPS_PORT>/jobhistory/logs
    yarn.resourcemanager.webapp.https.address=<RM>:<RM_HTTPS_PORT> 
    yarn.nodemanager.webapp.https.address=0.0.0.0:<NM_HTTPS_PORT>
  5. Create an ssl-server.xml file for the Hadoop SSL Keystore Factory:

    1. Copy the example SSL Server configuration file and modify the settings for your environment:

      cp /etc/hadoop/conf/ssl-server.xml.example /etc/hadoop/conf/ssl-server.xml
    2. Configure the server SSL properties:

      Table 4.3. Configuration Properties in ssl-server.xml
      Property Default Value Description
      ssl.server.keystore.type JKS The type of the keystore, JKS = Java Keystore, the de-facto standard in Java
      ssl.server.keystore.location None The location of the keystore file
      ssl.server.keystore.password None The password to open the keystore file
      ssl.server.truststore.type JKS The type of the trust store
      ssl.server.truststore.location None The location of the truststore file
      ssl server.truststore.password None The password to open the truststore

      For example:

      <property>
        <name>ssl.server.truststore.location</name>
        <value>/etc/security/serverKeys/truststore.jks</value>
        <description>Truststore to be used by NN and DN. Must be specified.</description>
      </property>
      
      <property>
        <name>ssl.server.truststore.password</name>
        <value>changeit</value>
        <description>Optional. Default value is "".
        </description>
      </property>
      
      <property>
        <name>ssl.server.truststore.type</name>
        <value>jks</value>
        <description>Optional. The keystore file format, default value is "jks".</description>
      </property>
      
      <property>
        <name>ssl.server.truststore.reload.interval</name>
        <value>10000</value>
        <description>Truststore reload check interval, in milliseconds.
        Default value is 10000 (10 seconds).</description>
      </property>
      
      <property>
        <name>ssl.server.keystore.location</name>
        <value>/etc/security/serverKeys/keystore.jks</value>
        <description>Keystore to be used by NN and DN. Must be specified.</description>
      </property>
      
      <property>
        <name>ssl.server.keystore.password</name>
        <value>changeit</value>
        <description>Must be specified.</description>
      </property>
      
      <property>
        <name>ssl.server.keystore.keypassword</name>
        <value>changeit</value>
        <description>Must be specified.</description>
      </property>
      
      <property>
        <name>ssl.server.keystore.type</name>
        <value>jks</value>
        <description>Optional. The keystore file format, default value is "jks".</description>
      </property>
  6. Create an ssl-client.xml file for the Hadoop SSL Keystore Factory:

    1. Copy the client truststore example file:

      cp /etc/hadoop/conf/ssl-server.xml.example /etc/hadoop/conf/ssl-server.xml
    2. Configure the client trust store values:

      ssl.client.truststore.location=/etc/security/clientKeys/all.jks
      ssl.client.truststore.password=clientTrustStorePassword
      ssl.client.truststore.type=jks
  7. Set the following properties in the tez-site.xml file:

    tez.runtime.shuffle.ssl.enable=true
    tez.runtime.shuffle.keep-alive.enabled=true
  8. Copy the configuration files (core-site.xml, hdfs-site.xml, mapred-site.xml, yarn-site.xml, ssl-server.xml, tez-site.xml and ssl-client.xml), including the ssl-server and ssl-client store files if the Hadoop SSL Keystore Factory uses it's own keystore and truststore files, to all nodes in the cluster.

  9. Restart services on all nodes in the cluster.