Configuring and Upgrading Apache Spark
Before you can upgrade Apache Spark, you must have first upgraded your HDP components to the latest version (in this case, 2.5.3). This section assumes that you have already upgraded your components for HDP 2.5.3. If you have not already completed these steps, return to Getting Ready to Upgrade and Upgrade 2.3 Components for instructions on how to upgrade your HDP components to 2.5.3.
To upgrade Spark, start the service and update configurations.
Stop the Spark
history-server. If you are using the Sparkthrift-server, stop thethrift-server.su - spark -c "$SPARK_HOME/sbin/stop-history-server.sh" su - spark -c "$SPARK_HOME/sbin/stop-thriftserver.sh"
Remove any reference to
hdp.versionfrom the Spark configuration files.Remove
spark.yarn.servicesproperty from spark-defaults.conf.Make sure that
spark.history.provider, if present, is set toorg.apache.spark.deploy.history.FsHistoryProvider.Restart the
history-server:su - spark -c "$SPARK_HOME/sbin/start-history-server.sh"
If you are using the Spark thrift-server, restart the
thrift-server. See (Optional) Starting the Spark Thrift Server.Validate the Spark installation. As user
spark, run the examples in Running Spark Applications in the Spark Guide.
For additional configuration information, see the Spark Guide.