Direct Reader configuration properties

You need to know the property names and valid values for configuring Direct Reader mode. The advantage of using Direct Reader V2 over Direct Reader V1 is its ability to process ORC data using vectorization, which improves performance.


In configuration/spark-defaults.conf, or using the --conf option in spark-submit/spark-shell set the following properties:

Name: spark.sql.extensions
Value: com.hortonworks.spark.sql.rule.Extensions

Required for using Spark SQL in auto-translate direct reader mode. Set before creating the spark session.

Required for using Direct Reader with the Spark Data Source API V1 or V2 (recommended), respectively.

Name: spark.kryo.registrator
Value: com.qubole.spark.hiveacid.util.HiveAcidKyroRegistrator
Set before the spark session. Required if serialization = kryo.
Name: spark.hadoop.hive.metastore.uris
Value: thrift://<host>:<port>
Hive metastore URI.
Name: --jars
Value: HWC jar
Pass the HWC jar to spark-shell or spark-submit using the --jars option while launching the application. For example, launch spark-shell as follows.

Example: Launch a spark-shell

spark-shell --jars \
    /opt/cloudera/parcels/CDH/lib/hive_warehouse_connector/hive-warehouse-connector-assembly-<version>.jar \
    --conf "spark.sql.extensions=com.hortonworks.spark.sql.rule.Extensions" \
    --conf "
    --conf "spark.kryo.registrator=com.qubole.spark.hiveacid.util.HiveAcidKyroRegistrator" \
    --conf "spark.hadoop.hive.metastore.uris=<metastore_uri>"