Cloudera Docs

Configuring the Spark Cluster

Configure the Spark cluster for FIPS environments by adding Cloudera Manager advanced configuration snippet settings to export SafeLogic modules and define Java cryptographic runtime options.

  1. Set the spark-env.sh advanced configuration snippet in Cloudera Manager for your deployed Spark role:
    • For SPARK_ON_YARN>GATEWAY: Add the spark-conf/spark-env.sh_client_config_safety_valve advanced configuration snippet settings for Spark cluster mode in the Cloudera Manager instance:
      export SPARK_SUBMIT_OPTS="$SPARK_SUBMIT_OPTS 
--add-exports=java.base/sun.security.provider=bctls 
--add-exports=java.base/sun.security.provider=com.safelogic.cryptocomply.fips.core 
--add-modules=com.safelogic.cryptocomply.fips.core,bctls 
--module-path=/opt/cloudera/fips"
    • For SPARK3_ON_YARN>GATEWAY: Add the spark3-conf/spark-env.sh_client_config_safety_valve advanced configuration snippet settings for Spark cluster mode in the Cloudera Manager instance:
      export SPARK_SUBMIT_OPTS="$SPARK_SUBMIT_OPTS 
--add-exports=java.base/sun.security.provider=bctls 
--add-exports=java.base/sun.security.provider=com.safelogic.cryptocomply.fips.core 
--add-modules=com.safelogic.cryptocomply.fips.core,bctls --module-path=/opt/cloudera/fips"
  2. Set the spark-defaults.conf advanced configuration snippet in Cloudera Manager to ensure Spark operates correctly in a FIPS environment:
    • For SPARK_ON_YARN>GATEWAY: Add settings to spark-conf/spark-defaults.conf_client_config_safety_valve.
      spark-conf/spark-defaults.conf_client_config_safety_valve
spark.yarn.am.extraJavaOptions=--add-exports=java.base/sun.security.provider=bctls 
--add-exports=java.base/sun.security.provider=com.safelogic.cryptocomply.fips.core 
--add-modules=com.safelogic.cryptocomply.fips.core,bctls --module-path=/opt/cloudera/fips 
-Dcom.safelogic.cryptocomply.fips.approved_only=true -Djava.net.preferIPv4Stack=true 
-Djdk.tls.ephemeralDHKeySize=2048 -Djdk.tls.trustNameService=true 
-Dorg.bouncycastle.jsse.client.assumeOriginalHostName=true
spark.driver.extraJavaOptions=--add-exports=java.base/sun.security.provider=bctls 
--add-exports=java.base/sun.security.provider=com.safelogic.cryptocomply.fips.core 
--add-modules=com.safelogic.cryptocomply.fips.core,bctls --module-path=/opt/cloudera/fips 
-Dcom.safelogic.cryptocomply.fips.approved_only=true -Djava.net.preferIPv4Stack=true 
-Djdk.tls.ephemeralDHKeySize=2048 -Djdk.tls.trustNameService=true 
-Dorg.bouncycastle.jsse.client.assumeOriginalHostName=true
spark.executor.extraJavaOptions=--add-exports=java.base/sun.security.provider=bctls 
--add-exports=java.base/sun.security.provider=com.safelogic.cryptocomply.fips.core 
--add-modules=com.safelogic.cryptocomply.fips.core,bctls --module-path=/opt/cloudera/fips 
-Dcom.safelogic.cryptocomply.fips.approved_only=true -Djava.net.preferIPv4Stack=true 
-Djdk.tls.ephemeralDHKeySize=2048 -Djdk.tls.trustNameService=true 
-Dorg.bouncycastle.jsse.client.assumeOriginalHostName=true
    • For SPARK3_ON_YARN>GATEWAY: Add settings to spark3-conf/spark-defaults.conf_client_config_safety_valve.
      spark.yarn.am.extraJavaOptions=--add-exports=java.base/sun.security.provider=bctls 
--add-exports=java.base/sun.security.provider=com.safelogic.cryptocomply.fips.core 
--add-modules=com.safelogic.cryptocomply.fips.core,bctls --module-path=/opt/cloudera/fips 
-Dcom.safelogic.cryptocomply.fips.approved_only=true -Djava.net.preferIPv4Stack=true 
-Djdk.tls.ephemeralDHKeySize=2048 -Djdk.tls.trustNameService=true 
-Dorg.bouncycastle.jsse.client.assumeOriginalHostName=true
spark.driver.defaultJavaOptions=--add-exports=java.base/sun.security.provider=bctls 
--add-exports=java.base/sun.security.provider=com.safelogic.cryptocomply.fips.core 
--add-modules=com.safelogic.cryptocomply.fips.core,bctls --module-path=/opt/cloudera/fips 
-Dcom.safelogic.cryptocomply.fips.approved_only=true -Djava.net.preferIPv4Stack=true 
-Djdk.tls.ephemeralDHKeySize=2048 -Djdk.tls.trustNameService=true 
-Dorg.bouncycastle.jsse.client.assumeOriginalHostName=true
spark.executor.defaultJavaOptions=--add-exports=java.base/sun.security.provider=bctls 
--add-exports=java.base/sun.security.provider=com.safelogic.cryptocomply.fips.core 
--add-modules=com.safelogic.cryptocomply.fips.core,bctls
      .