Configuring Phoenix-Spark connector using Cloudera Manager when both are on same cluster

Learn how to configure the Phoenix-Spark connector when both the Phoenix and Spark are on the same cluster.

  1. Go to the Spark or Spark3 service.
  2. Click the Configuration tab.
  3. Ensure that the HBase service is selected in Spark Service as a dependency.
  4. Select Scope > Gateway.
  5. Select Category > Advanced.
  6. Locate the Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-defaults.conf property or search for it by typing its name in the Search box.
  7. Add the following properties to ensure that all required Phoenix and HBase platform dependencies are available on the classpath for the Spark executors and drivers:

    Phoenix client JARs for Spark2:

    spark.executor.extraClassPath=/opt/cloudera/parcels/CDH/lib/phoenix_connectors/phoenix5-spark-shaded.jar
    spark.driver.extraClassPath=/opt/cloudera/parcels/CDH/lib/phoenix_connectors/phoenix5-spark-shaded.jar

    If you are using the Phoenix Spark3 pursers, consider the following client JARs:

    spark.executor.extraClassPath=/opt/cloudera/parcels/SPARK3/lib/spark3/phoenix_connectors/phoenix5-spark3-shaded.jar
    spark.driver.extraClassPath=/opt/cloudera/parcels/SPARK3/lib/spark3/phoenix_connectors/phoenix5-spark3-shaded.jar
  8. Enter a Reason for change, and then click Save Changes to commit the changes.
  9. Restart the role and service when Cloudera Manager prompts you to restart.
  • Build a Spark or Spark3 application using the dependencies that you provide when you run your application. If you follow the previous instructions, Cloudera Manager automatically configures the connector. If you have not, add the necessary parameters to the command line when running the spark-submit or spark3-submit command.
    --conf spark.executor.extraClassPath=/opt/cloudera/parcels/SPARK3/lib/spark3/phoenix_connectors/phoenix5-spark3-shaded.jar --conf spark.driver.extraClassPath=/opt/cloudera/parcels/SPARK3/lib/spark3/phoenix_connectors/phoenix5-spark3-shaded.jar
  • Consider the following example while using a Spark3 application.
    spark3-shell --conf spark.executor.extraClassPath=/opt/cloudera/parcels/SPARK3/lib/spark3/hbase_connectors/lib/hbase-spark3-***VERSION NUMBER***.jar:/opt/cloudera/parcels/SPARK3/lib/spark3/hbase_connectors/lib/hbase-spark3-protocol-shaded-***VERSION NUMBER***.jar:/opt/cloudera/parcels/SPARK3/lib/spark3/phoenix_connectors/phoenix5-spark3-shaded.jar:/etc/hbase/conf/ --conf spark.driver.extraClassPath=/opt/cloudera/parcels/SPARK3/lib/spark3/hbase_connectors/lib/hbase-spark3-***VERSION NUMBER***.jar:/opt/cloudera/parcels/SPARK3/lib/spark3/hbase_connectors/lib/hbase-spark3-protocol-shaded-***VERSION NUMBER***.jar:/opt/cloudera/parcels/SPARK3/lib/spark3/phoenix_connectors/phoenix5-spark3-shaded.jar:/etc/hbase/conf/