Use HWC/Spark Direct Reader for Spark Apps/ETL

You need to know a little about Hive Warehouse Connector (HWC) and how to find more information because to access Hive from Spark, you need to use HWC implicitly or explicitly.

HWC is a Spark library/plugin that is launched with the Spark app. Use the Spark Direct Reader and HWC for ETL.

The Hive Warehouse Connector is designed to access managed ACID v2 Hive tables from Spark. Apache Ranger and the HiveWarehouseConnector library provide row and column, fine-grained access to the data. HWC supports spark-submit and pyspark. The spark thrift server is not supported.