Cloudera Docs

Step 5: Configure the Spark Cluster

Additional configuration required while working with the Spark cluster.

  1. In your Cloudera Manager instance, add the following advanced configuration snippet settings for the Spark cluster mode:
    For the SPARK_ON_YARN>GATEWAY role:
    spark-conf/spark-env.sh_client_config_safety_valve
    export SPARK_SUBMIT_OPTS="$SPARK_SUBMIT_OPTS 
--add-exports=java.base/sun.security.provider=bctls 
--add-exports=java.base/sun.security.provider=com.safelogic.cryptocomply.fips.core 
--add-modules=com.safelogic.cryptocomply.fips.core,bctls 
--module-path=<BCTLS_JARS_DIR>"
    For the SPARK3_ON_YARN > GATEWAY role:
    spark3-conf/spark-env.sh_client_config_safety_valve
    export SPARK_SUBMIT_OPTS="$SPARK_SUBMIT_OPTS 
--add-exports=java.base/sun.security.provider=bctls 
--add-exports=java.base/sun.security.provider=com.safelogic.cryptocomply.fips.core 
--add-modules=com.safelogic.cryptocomply.fips.core,bctls 
--module-path=<BCTLS_JARS_DIR>"
    <BCTLS_JARS_DIR> is the directory containing the SafeLogic bctls and fips core jar files.
  2. For Spark to work correctly on FIPS, add the following advanced configuration snippet settings:
    For the SPARK_ON_YARN>GATEWAY role:
    spark-conf/spark-defaults.conf_client_config_safety_valve
    spark.yarn.am.extraJavaOptions=--add-exports=java.base/sun.security.provider=bctls 
--add-exports=java.base/sun.security.provider=com.safelogic.cryptocomply.fips.core 
--add-modules=com.safelogic.cryptocomply.fips.core,bctls --module-path=<BCTLS_JARS_DIR> 
-Dcom.safelogic.cryptocomply.fips.approved_only=true -Djava.net.preferIPv4Stack=true 
-Djdk.tls.ephemeralDHKeySize=2048 -Djdk.tls.trustNameService=true 
-Dorg.bouncycastle.jsse.client.assumeOriginalHostName=true
spark.driver.extraJavaOptions=--add-exports=java.base/sun.security.provider=bctls 
--add-exports=java.base/sun.security.provider=com.safelogic.cryptocomply.fips.core 
--add-modules=com.safelogic.cryptocomply.fips.core,bctls --module-path=<BCTLS_JARS_DIR> 
-Dcom.safelogic.cryptocomply.fips.approved_only=true -Djava.net.preferIPv4Stack=true 
-Djdk.tls.ephemeralDHKeySize=2048 -Djdk.tls.trustNameService=true 
-Dorg.bouncycastle.jsse.client.assumeOriginalHostName=true
spark.executor.extraJavaOptions=--add-exports=java.base/sun.security.provider=bctls 
--add-exports=java.base/sun.security.provider=com.safelogic.cryptocomply.fips.core 
--add-modules=com.safelogic.cryptocomply.fips.core,bctls --module-path=<BCTLS_JARS_DIR> 
-Dcom.safelogic.cryptocomply.fips.approved_only=true -Djava.net.preferIPv4Stack=true 
-Djdk.tls.ephemeralDHKeySize=2048 -Djdk.tls.trustNameService=true 
-Dorg.bouncycastle.jsse.client.assumeOriginalHostName=true
    For the SPARK3_ON_YARN > GATEWAY role:
    spark3-conf/spark-defaults.conf_client_config_safety_valve
    spark.yarn.am.extraJavaOptions=--add-exports=java.base/sun.security.provider=bctls 
--add-exports=java.base/sun.security.provider=com.safelogic.cryptocomply.fips.core 
--add-modules=com.safelogic.cryptocomply.fips.core,bctls --module-path=<BCTLS_JARS_DIR> 
-Dcom.safelogic.cryptocomply.fips.approved_only=true -Djava.net.preferIPv4Stack=true 
-Djdk.tls.ephemeralDHKeySize=2048 -Djdk.tls.trustNameService=true 
-Dorg.bouncycastle.jsse.client.assumeOriginalHostName=true
spark.driver.defaultJavaOptions=--add-exports=java.base/sun.security.provider=bctls 
--add-exports=java.base/sun.security.provider=com.safelogic.cryptocomply.fips.core 
--add-modules=com.safelogic.cryptocomply.fips.core,bctls --module-path=<BCTLS_JARS_DIR> 
-Dcom.safelogic.cryptocomply.fips.approved_only=true -Djava.net.preferIPv4Stack=true 
-Djdk.tls.ephemeralDHKeySize=2048 -Djdk.tls.trustNameService=true 
-Dorg.bouncycastle.jsse.client.assumeOriginalHostName=true
spark.executor.defaultJavaOptions=--add-exports=java.base/sun.security.provider=bctls 
--add-exports=java.base/sun.security.provider=com.safelogic.cryptocomply.fips.core 
--add-modules=com.safelogic.cryptocomply.fips.core,bctls 
--module-path=<BCTLS_JARS_DIR> 
-Dcom.safelogic.cryptocomply.fips.approved_only=true -Djava.net.preferIPv4Stack=true 
-Djdk.tls.ephemeralDHKeySize=2048 -Djdk.tls.trustNameService=true 
-Dorg.bouncycastle.jsse.client.assumeOriginalHostName=true

    <BCTLS_JARS_DIR> is the directory containing the SafeLogic bctls and fips core jar files.