Direct Reader configuration properties

You need to know the property names and valid values for configuring Direct Reader mode. The advantage of using Direct Reader V2 over Direct Reader V1 is its ability to process ORC data using vectorization, which improves performance.

Options

In configuration/spark-defaults.conf, or using the --conf option in spark-submit/spark-shell set the following properties:

Name: spark.sql.extensions: Value: com.hortonworks.spark.sql.rule.Extensions; Required for using Spark SQL in auto-translate direct reader mode. Set before creating the spark session.
spark.datasource.hive.warehouse.read.mode: DIRECT_READER_V1 or DIRECT_READER_V2; Required for using Direct Reader with the Spark Data Source API V1 or V2 (recommended), respectively.

Name: spark.kryo.registrator: Value: com.qubole.spark.hiveacid.util.HiveAcidKyroRegistrator; Set before the spark session. Required if serialization = kryo.

Name: spark.hadoop.hive.metastore.uris: Value: thrift://<host>:<port>; Hive metastore URI.

Name: --jars: Value: HWC jar; Pass the HWC jar to spark-shell or spark-submit using the --jars option while launching the application. For example, launch spark-shell as follows.

Example: Launch a spark-shell

spark-shell --jars \
    /opt/cloudera/parcels/CDH/lib/hive_warehouse_connector/hive-warehouse-connector-assembly-<version>.jar \
    --conf "spark.sql.extensions=com.hortonworks.spark.sql.rule.Extensions" \
    --conf "spark.datasource.hive.warehouse.read.mode=DIRECT_READER_V2
    --conf "spark.kryo.registrator=com.qubole.spark.hiveacid.util.HiveAcidKyroRegistrator" \
    --conf "spark.hadoop.hive.metastore.uris=<metastore_uri>"