Direct Reader mode introduction

Direct Reader mode is a transparent connection that Hive Warehouse Connector (HWC) makes to Apache Hive metastore (HMS) to get transaction information. In Direct Reader mode, Spark reads the data directly from the managed table location using the transaction snapshot. You use this mode if you do not need production-level Ranger authorization. Direct Reader mode does not support Ranger authorization.

Requirements and recommendations

Spark Direct Reader mode requires a connection to Hive metastore. Neither HiveServer (HS2) nor HiveServer Interactive (HSI) connections are needed.

Spark Direct Reader for reading Hive ACID, transactional tables from Spark is supported for production use. Use Spark Direct Reader mode if your ETL jobs do not require authorization and run as super user.

Component interaction

The following diagram shows component interaction ifor Direct Reader reads.