Upgrading HDP Manually
Also available as:
loading table of contents...

Configuring and Upgrading Apache Spark

  1. Add the node where you want Apache Spark 1.4.1 History Server to run. Install the version corresponding to the HDP version you currently have installed.

    1. su - root

    2. wget -nv http://s3.amazonaws.com/dev.hortonworks.com/ HDP/centos6/2.x/BUILDS/<version>/hdpbn.repo -O /etc/ yum.repos.d/Spark<version>.repo

    3. yum install spark_<version>-master -y

    4. To use Python: yum install spark_<version>-python

    5. conf-select create-conf-dir --package spark --stack-version <version> --conf-version 0

    6. cp /etc/spark/<version>/0/* /etc/spark/<version>/0/

    7. conf-select set-conf-dir --package spark --stack-version <version> --conf-version 0

    8. hdp-select set spark-client <version>

    9. hdp-select set spark-historyserver <version>

  2. Validate the Spark installation. As user spark, run SparkPI example:

    1. sudo su spark

    2. cd /usr/hdp/current/spark-client

    3. ./bin/run-example SparkPi 10

  3. Restart Spark on YARN in either yarn-cluster mode or yarn-client mode:

    • yarn-cluster mode: ./usr/hdp/current/spark-client/bin/spark-submit --class path.to.your.Class --master yarn-cluster [options] [app options]

    • yarn-client mode: ./usr/hdp/current/spark-client/bin/spark-shell -- master yarn-client