HDP-2.3.4 Release Notes
Also available as:

Optional: Spark Manual Upgrade Procedure


Hortonworks strongly recommends that all users running HDP 2.3.4 upgrade to HDP

(Optional) Upgrade Spark from 1.4.1 to 1.5.2. As root:

  1. Stop Spark 1.4.1: su - spark -c "/usr/hdp/current/spark-client/sbin/stop-history-server.sh".

  2. Remove Spark 1.4.1: yum erase "spark*".

  3. Add the node where you want Spark 1.5.2 History Server to run:

    1. su - root

    2. wget -nv http://s3.amazonaws.com/dev.hortonworks.com/HDP/centos6/2.x/BUILDS/ -O /etc/yum.repos.d/Spark141TP.repo

    3. yum install spark_2_3_4_0_3485-master -y

    4. To use Python: yum install spark_2_3_4_0_3485-python

    5. conf-select create-conf-dir --package spark --stack-version --conf-version 0

    6. cp /etc/spark/* /etc/spark/

    7. conf-select set-conf-dir --package spark --stack-version --conf-version 0

    8. hdp-select set spark-client

    9. hdp-select set spark-historyserver

  4. Validate the Spark installation. As user spark, run SparkPI example:

    1. su - spark -c "cd /usr/hdp/current/spark-client"

    2. ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --num-executors 3 --driver-memory 512m --executor-memory 512m --executor-cores 1 lib/spark-examples*.jar 10

  5. Restart Spark on YARN in either yarn-cluster mode or yarn-client mode:

    • yarn-cluster mode: ./usr/hdp/current/spark-client/bin/spark-submit --class path.to.your.Class --master yarn-cluster [options] <app jar> [app options]

    • yarn-client mode: ./usr/hdp/current/spark-client/bin/spark-shell --master yarn-client