Considerations for setting up spark
Set up Spark based on your requirement. Following are some of the considerations that you will have to take into account.
-
You should configure the 'spark.executor.extraClassPath' and ‘spark.driver.extraClassPath’ in spark-defaults.conf file to include the ‘phoenix-<version>-client.jar’ to ensure that all required Phoenix and HBase platform dependencies are available on the classpath for the Spark executors and drivers.
-
HDP Version Spark Version JARs to add (order dependent) >=2.6.2 (including 3.0.0) Spark 2 phoenix-<version>-spark2.jar
phoenix-<version>-client.jar
>=2.6.2 (including 3.0.0) Spark 1 phoenix-<version>-spark.jar
phoenix-<version>-client.jar
2.6.0-2.6.1 Spark 2 Unsupported: upgrade to at least HDP-2.6.2 2.6.0-2.6.1 Spark 1 phoenix-<version>-spark.jar
phoenix-<version>-client.jar
2.5.x Spark 1 phoenix-<version>-client-spark.jar
-
To enable your IDE, you can add the following provided dependency to your build:
<dependency><groupId>org.apache.phoenix</groupId> <artifactId>phoenix-spark</artifactId> <version>${phoenix.version}</version> <scope>provided</scope></dependency>