Optional: Spark Manual Upgrade Procedure

Spark 1.3.1 is newly available as of HDP 2.2.8. You can choose to upgrade Spark from 1.2.1 to 1.3.1, or continue using Spark 1.2.1.

If you wish to upgrade to Spark 1.3.1, perform the following steps as root:

Install Spark:

Enter wget -nv http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.2.9.0/hdp.repo -O /etc/yum.repos.d/hdp229.repo.

Enter yum search spark.

The prompt returns:

spark.noarch : spark Distro virtual package
spark-master.noarch : spark-master Distro virtual package
spark-python.noarch : spark-python Distro virtual package
spark-worker.noarch : spark-worker Distro virtual package
spark_2_2_9_0_3393-master.noarch : Server for Spark master
spark_2_2_9_0_3393-python.noarch : Python client for Spark
spark_2_2_9_0_3393-worker.noarch : Server for Spark worker
spark_2_2_9_0_3393-master.noarch : Server for Spark master
spark_2_2_9_0_3393-python.noarch : Python client for Spark
spark_2_2_9_0_3393-worker.noarch : Server for Spark worker
spark_2_2_9_0_3393.noarch : Lightning-Fast Cluster Computing
spark_2_2_9_0_3393.noarch : Lightning-Fast Cluster Computing

To install Spark 1.3.1, select spark_2_2_9_0_3393: yum install spark_2_2_9_0_3393-master.
Enter hdp-select set spark-client 2.2.9.0-3393.
Enter hdp-select set spark-historyserver 2.2.9.0-3393.

Validate the Spark installation:
1. Enter su spark
2. Enter cd /usr/hdp/current/spark-client.
3. Enter ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --num-executors 3 --driver-memory 512m --executor-memory 512m --executor-cores 1 lib/spark-examples*.jar 10.
Optional: Run Spark History Server:
1. Enter su spark.
2. Enter cd /usr/hdp/current/spark-historyserver/.
3. Enter ./sbin/start-history-server.sh.

​Optional: Spark Manual Upgrade Procedure

Optional: Spark Manual Upgrade Procedure