Replace the hdp version in $SPARK_HOME/conf/spark-defaults.conf and $SPARK_HOME/conf/java-opts with the current hadoop version.

su - spark -c "$SPARK_HOME/sbin/start-history-server.sh"

Restart the history server:

su - spark -c "usr/hdp/current//start-historyserver/sbin/start-history-server.sh"

If you will be running Spark in yarn-client mode, update the following property in /etc/hadoop/conf/mapred-site.xml by substituting ${hdp.version} with the actual HDP version (2.3.2.0.0- $version>).

  
  <property>
  <name>mapreduce.application.classpath</name> 
    <value>$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*,
      $PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*,
      $PWD/mr-framework/hadoop/share/hadoop/common/*,
      $PWD/mr-framework/hadoop/share/hadoop/common/lib/*,
      $PWD/mr-framework/hadoop/share/hadoop/yarn/*,
      $PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*,
      $PWD/mr-framework/hadoop/share/hadoop/hdfs/*,
      $PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*,
      /usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar,
      /etc/hadoop/conf/secure</value>
  </property>

Restart Spark on YARN in either yarn-cluster mode or yarn-client mode:

yarn-cluster mode:

/usr/hdp/current/spark-client/bin/spark-submit --class path.to.your.Class --master yarn-cluster [options] <app jar> [app
    options]

yarn-client mode:

    
./usr/hdp/current/spark-client/bin/spark-shell --master yarn-client
 

To enable Spark to work in LzoCodec, add the following information to /etc/spark/conf/spark-defaults.conf, when running on client-mode:

spark.driver.extraLibraryPath /usr/hdp/current/hadoop-client/lib/naive:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64

spark.driver.extraClassPath /usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.3.2.0.0-2492.jar