Direct Reader configuration properties

You need to know the property names and valid values for configuring Direct Reader mode.

Options

In configuration/spark-defaults.conf, or using the --conf option in spark-submit/spark-shell set the following properties:

Name: spark.sql.extensions: Value: com.hortonworks.spark.sql.rule.Extensions; Required for using Spark SQL in auto-translate direct reader mode. Set before creating the spark session.
spark.datasource.hive.warehouse.read.mode: DIRECT_READER_V1; Required for performing Direct Reader reads.

Name: spark.kryo.registrator: Value: com.qubole.spark.hiveacid.util.HiveAcidKyroRegistrator; Set before the spark session. Required if serialization = kryo.

Name: spark.hadoop.hive.metastore.uris: Value: thrift://<host>:<port>; Hive metastore URI.

Name: --jars: Value: HWC jar; Pass the HWC jar to spark-shell or spark-submit using the --jars option while launching the application. For example, launch spark-shell as follows.

Example: Launch a spark-shell

spark-shell --jars \
    /opt/cloudera/parcels/CDH/lib/hive_warehouse_connector/hive-warehouse-connector-assembly-<version>.jar \
    --conf "spark.sql.extensions=com.hortonworks.spark.sql.rule.Extensions" \
    --conf "spark.datasource.hive.warehouse.read.mode=DIRECT_READER_V1
    --conf "spark.kryo.registrator=com.qubole.spark.hiveacid.util.HiveAcidKyroRegistrator" \
    --conf "spark.hadoop.hive.metastore.uris=<metastore_uri>"