Cloudera Docs

Using Spark Hive Warehouse and HBase Connector Client .jar files with Livy

This section describes how to use Spark Hive Warehouse Connector (HWC) and Spark HBase Connector (SHC) client .jar files with Livy. These steps are required to ensure token acquisition and avoid authentication errors.

Use the following steps to use Spark HWC and SHC client .jar files with Livy:

  1. Copy the applicable HWC or SHC .jar files to the Livy server node and add these folders to the livy.file.local-dir-whitelist property in the livy.conf file.
  2. Add the required configurations in the /usr/hdp/current/spark2-client/conf folder:
    • For Hive, in /usr/hdp/current/spark2-client/conf/hive-site.xml
    • For HBase, in /usr/hdp/current/spark2-client/conf/hbase-site.xml).
    Or add the required configurations using the conf field in the session creation request. This is equivalent to using "--conf" in spark-submit.
  3. Reference these local .jar files in the session creation request using the file:/// URI format.

HWC Example

  1. Add the add the following folders to the livy.file.local-dir-whitelist property in the livy.conf file.
    /usr/hdp/current/hive_warehouse_connector/
  2. Add hive-site.xml to /usr/hdp/current/spark2-client/conf on all cluster nodes.
  3. When running using the Zeppelin Livy interpreter, reference the HWC .jar file as shown below. 
    %livy2.conf
livy.spark.jars file:///usr/hdp/current/hive_warehouse_connector/hive-warehouse-connector-assembly-1.0.0.3.0.0.0-1634.jar

SHC Example

  1. Add the add the following folders to the livy.file.local-dir-whitelist property in the livy.conf file.
    /usr/hdp/current/hbase-client/lib, /usr/hdp/current/shc
  2. Add hbase-site.xml to /usr/hdp/current/spark2-client/conf on all cluster nodes.
  3. When running using the Zeppelin Livy interpreter, reference the following HBase .jar files as shown below. Note that some of these .jar files have 644/root permissions, and therefore may throw an exception. If this happens, you may need to change the permissions of the applicable .jar files on the Livy node.
    %livy2.conf
livy.spark.jars file:///usr/hdp/current/shc/shc-core-1.1.0.3.0.1.0-65.jar, 
file:///usr/hdp/current/hbase-client/lib/hbase-shaded-protobuf-2.1.0.jar, 
file:///usr/hdp/current/hbase-client/lib/hbase-shaded-miscellaneous-2.1.0.jar, 
file:///usr/hdp/current/hbase-client/lib/hbase-protocol-shaded.jar, 
file:///usr/hdp/current/hbase-client/lib/hbase-shaded-netty-2.1.0.jar, 
file:///usr/hdp/current/hbase-client/lib/hbase-shaded-client.jar,  
file:///usr/hdp/current/hbase-client/lib/hbase-shaded-mapreduce.jar, 
file:///usr/hdp/current/hbase-client/lib/hbase-common.jar, 
file:///usr/hdp/current/hbase-client/lib/hbase-server.jar, 
file:///usr/hdp/current/hbase-client/lib/hbase-client.jar, 
file:///usr/hdp/current/hbase-client/lib/hbase-protocol.jar, 
file:///usr/hdp/current/hbase-client/lib/hbase-mapreduce.jar, 
file:///usr/hdp/current/hbase-client/lib/guava-11.0.2.jar