Configure Phoenix-Spark connector using Cloudera Manager
Learn how to configure Phoenix-Spark connector using Cloudera Manager.
Build a Spark application using the dependencies that you provide when you run your application. If you follow the previous instructions, Cloudera Manager automatically configures the connector for Spark. If you have not:
Consider the following example while using a Spark
application:
spark3-shell --conf spark.executor.extraClassPath=/opt/cloudera/parcels/CDHlib/phoenix_connectors/phoenix5-spark3-shaded-6.x.y.VERSION.jar --conf spark.jars=hdfs:///path/hbase_jars_common/hbase-site.xml.jar,hdfs:///path/hbase_jars_spark3/hbase-spark3-protocol-shaded.jar,hdfs:///path/hbase_jars_spark3/hbase-spark3.jar,hdfs:///path/hbase_jars_common/hbase-shaded-mapreduce-***VERSION NUMBER***.jar,hdfs:///path/hbase_jars_common/opentelemetry-api-***VERSION NUMBER***.jar,hdfs:///path/hbase_jars_common/opentelemetry-context-***VERSION NUMBER***.jar,hdfs:////path/hbase_jars_spark3/phoenix5-spark3-shaded.jar
If you run the spark3-submit command in the Yarn cluster mode, update the example as below.
spark3-submit --conf spark.driver.extraClassPath=/opt/cloudera/parcels/CDHlib/phoenix_connectors/phoenix5-spark3-shaded-6.x.y.VERSION.jar --conf spark.executor.extraClassPath=/opt/cloudera/parcels/CDHlib/phoenix_connectors/phoenix5-spark3-shaded-6.x.y.VERSION.jar spark.jars=hdfs:///path/hbase_jars_common/hbase-site.xml.jar,hdfs:///path/hbase_jars_spark3/hbase-spark3-protocol-shaded.jar,hdfs:///path/hbase_jars_spark3/hbase-spark3.jar,hdfs:///path/hbase_jars_common/hbase-shaded-mapreduce**VERSION NUMBER*.jar,hdfs:///path/hbase_jars_common/opentelemetry-api-*VERSION NUMBER*.jar,hdfs:///path/hbase_jars_common/opentelemetry-context-*VERSION NUMBER**.jar,hdfs:////path/hbase_jars_spark3/phoenix5-spark3-shaded.jar"