Direct Reader configuration properties

You need to know the property names and valid values for configuring Direct Reader mode.

Options

In configuration/spark-defaults.conf, or using the --conf option in spark-submit/spark-shell set the following properties:

Name: spark.sql.extensions
Value: com.hortonworks.spark.sql.rule.Extensions

Required for using Spark SQL in auto-translate direct reader mode. Set before creating the spark session.

spark.datasource.hive.warehouse.read.mode
DIRECT_READER_V1
Required for performing Direct Reader reads.
Name: spark.kryo.registrator
Value: com.qubole.spark.hiveacid.util.HiveAcidKyroRegistrator
Set before the spark session. Required if serialization = kryo.
Name: spark.hadoop.hive.metastore.uris
Value: thrift://<host>:<port>
Hive metastore URI.
Name: --jars
Value: HWC jar
Pass the HWC jar to spark-shell or spark-submit using the --jars option while launching the application. For example, launch spark-shell as follows.

Example: Launch a spark-shell

spark-shell --jars \
    /opt/cloudera/parcels/CDH/lib/hive_warehouse_connector/hive-warehouse-connector-assembly-<version>.jar \
    --conf "spark.sql.extensions=com.hortonworks.spark.sql.rule.Extensions" \
    --conf "spark.datasource.hive.warehouse.read.mode=DIRECT_READER_V1
    --conf "spark.kryo.registrator=com.qubole.spark.hiveacid.util.HiveAcidKyroRegistrator" \
    --conf "spark.hadoop.hive.metastore.uris=<metastore_uri>"