Compatible storage formats

You need to know the recommended formats for Hive ACID tables, and how you can access tables from Spark. The HMS translation layer checks the capabilities of a Hive client that tries to access Hive and returns an error message designed to help you resolve an access problem.

For Hive ACID, ORC is the recommended native storage format. You can insert, append, update, delete data in ORC format, but Hive ACID is not restricted to ORC. You can do insert/append operations to files in most other formats, such as text, Parquet, and AVRO. For more information about ORC, see ORC file format and Advanced ORC properties in Hive Performance Tuning.

Spark is not natively compatible with ACID tables. You need to use Hive Warehouse Connector (HWC) to read Hive ACID tables from the Hive metastore. There are two modes for HWC: JDBC mode and Direct Reader mode. The HMS translation layer prevents Spark from accessing Hive tables.

The HMS translation layer checks each client connection to determine the capabilities of the client. For example, HMS checks whether or not the client supports ORC and transactional tables. When a Spark client talks to the metastore, it can bypass HiveServer (HS2). Some operations, such as Direct Reader and Hive Streaming, go to Hive directly through HMS and reveal its capabilities to the HMS translation layer. If Spark does not have a connection to HWC associated, when the Spark user tries to get information about the ACID table, the query fails. The HMS translation layer determines that Spark without HWC does not have the required capabilities to access ACID tables, and gives you an error.