Configuring Phoenix-Spark connector using Cloudera Manager when both are on same cluster
Learn how to configure the Phoenix-Spark connector when both the Phoenix and Spark are on the same cluster.
- Build a Spark or Spark3 application using the dependencies that you provide when
you run your application. If you follow the previous instructions, Cloudera
Manager automatically configures the connector. If you have not, add the
necessary parameters to the command line when running the
spark-submit or spark3-submit command.
--conf spark.executor.extraClassPath=/opt/cloudera/parcels/SPARK3/lib/spark3/phoenix_connectors/phoenix5-spark3-shaded.jar --conf spark.driver.extraClassPath=/opt/cloudera/parcels/SPARK3/lib/spark3/phoenix_connectors/phoenix5-spark3-shaded.jar
- Consider the following example while using a Spark3
application.
spark3-shell --conf spark.executor.extraClassPath=/opt/cloudera/parcels/SPARK3/lib/spark3/hbase_connectors/lib/hbase-spark3-***VERSION NUMBER***.jar:/opt/cloudera/parcels/SPARK3/lib/spark3/hbase_connectors/lib/hbase-spark3-protocol-shaded-***VERSION NUMBER***.jar:/opt/cloudera/parcels/SPARK3/lib/spark3/phoenix_connectors/phoenix5-spark3-shaded.jar:/etc/hbase/conf/ --conf spark.driver.extraClassPath=/opt/cloudera/parcels/SPARK3/lib/spark3/hbase_connectors/lib/hbase-spark3-***VERSION NUMBER***.jar:/opt/cloudera/parcels/SPARK3/lib/spark3/hbase_connectors/lib/hbase-spark3-protocol-shaded-***VERSION NUMBER***.jar:/opt/cloudera/parcels/SPARK3/lib/spark3/phoenix_connectors/phoenix5-spark3-shaded.jar:/etc/hbase/conf/