Submit a Scala or Java application
A step-by-step procedure shows you how to submit an app based on the HiveWarehouseConnector library to run on Apache Spark Shell.
- Choose an execution mode, for example the HWC JDBC execution mode, for your application and check that you meet the configuration requirements, described earlier.
- Configure a Spark-HiveServer connection, described earlier or, in your app submission include the appropriate --conf in step 4.
Locate the hive-warehouse-connector-assembly jar in the
/hive_warehouse_connector/directory.For example, find
hive-warehouse-connector-assembly-220.127.116.11.2.1.0-327.jarin the following location:
Add the connector jar and configurations to the app submission using the
spark-shell --jars <path to jars>/hive_warehouse_connector/hive-warehouse-connector-assembly-<version>.jar \ --conf <configuration properties>
Add the path to app you wrote based on the HiveWarehouseConnector API.
<path to app>
spark-shell --jars /opt/cloudera/parcels/CDH/jars/hive-warehouse-connector-assembly-18.104.22.168.2.1.0-327.jar \ --conf spark.sql.hive.hwc.execution.mode=spark \ --conf spark.datasource.hive.warehouse.read.via.llap=false \ --conf spark.datasource.hive.warehouse.load.staging.dir=<path to directory> \ /home/myapps/myapp.jarPySpark and spark-submit are also supported.