DocsHortonworks Data Platform

 

 3.2. Optional: Spark Manual Upgrade Procedure

Spark 1.3.1 is newly available in HDP 2.2.8. You can choose to upgrade Spark from 1.2.1 to 1.3.1, or continue using Spark 1.2.1.

If you wish to upgrade to Spark 1.3.1, perform the following steps as root:

  1. Install Spark:

    1. Enter wget -nv http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.2.8.0/hdp.repo -O /etc/yum.repos.d/hdp228.repo.

    2. Enter yum search spark.

      The prompt returns:

      spark.noarch : spark Distro virtual package
spark-master.noarch : spark-master Distro virtual package
spark-python.noarch : spark-python Distro virtual package
spark-worker.noarch : spark-worker Distro virtual package
spark_2_2_8_0_3150-master.noarch : Server for Spark master
spark_2_2_8_0_3150-python.noarch : Python client for Spark
spark_2_2_8_0_3150-worker.noarch : Server for Spark worker
spark_2_2_8_0_3150-master.noarch : Server for Spark master
spark_2_2_8_0_3150-python.noarch : Python client for Spark
spark_2_2_8_0_3150-worker.noarch : Server for Spark worker
spark_2_2_8_0_3150.noarch : Lightning-Fast Cluster Computing
spark_2_2_8_0_3150.noarch : Lightning-Fast Cluster Computing

    3. To install Spark 1.3.1, select spark_2_2_8_0_3150: yum install spark_2_2_8_0_3150-master.

    4. Enter hdp-select set spark-client 2.2.8.0-3150.

    5. Enter hdp-select set spark-historyserver 2.2.8.0-3150.

  2. Validate the Spark installation:

    1. Enter su spark

    2. Enter cd /usr/hdp/current/spark-client.

    3. Enter ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --num-executors 3 --driver-memory 512m --executor-memory 512m --executor-cores 1 lib/spark-examples*.jar 10.

  3. Optional: Run Spark History Server:

    1. Enter su spark.

    2. Enter cd /usr/hdp/current/spark-historyserver/.

    3. Enter ./sbin/start-history-server.sh.

Legal notices