HMS table storage

You need to understand how Hive metastore (HMS) stores Hive tables when you run a CREATE TABLE statement or migrate a table to HDP during an upgrade from an HDP 2.6 cluster. The success or failure of the statement, the resulting table type, and the table location depends on a number of factors.

HMS table transformations

The HMS includes the following Hive metadata about tables that you create:

A table definition
Column names
Data types
Comments in a central schema repository

When you use EXTERNAL keyword in the CREATE TABLE statement, HMS stores the table as an external table. When you omit the EXTERNAL keyword and create a managed table, or ingest a managed table, HMS might translate the table into an external table or the table creation can fail, depending on the table properties. An important table property that affects table transformations is the ACID or Non-ACID table type:

Non-ACID: Table properties do not contain any ACID related properties set to true. For example, the table does not contain such properties transactional=true or insert_only=true.

ACID: Table properties do contain one or more ACID properties set to true.

Full ACID: Table properties contain transactional=true but not insert_only=true.

Insert-only ACID: Table properties contain insert_only=true.

The following matrix shows the table type and whether or not the location property is supported.


ACID	Managed	Location Property	Comments	Action
Non-ACID	Yes	Yes	Migrated from an HDP 2.6 cluster	Table stored as external
Non-ACID, ACID, full ACID, insert-only ACID	Yes	No	Table location is null	Table stored in subdirectory `metastore.warehouse.external.dir`

HMS detects type of client for interacting with HMS, for example Hive or Spark, and compares the capabilities of the client with the table requirement. HMS performs the following actions, depending on the results of the comparison:


Table requirement	Client meets requirements	Managed Table	ACID table type	Action
Client can write to any type of ACID table	No	Yes	Yes	CREATE TABLE fails
Client can write to full ACID table	No	Yes	insert_only=true	CREATE TABLE fails
Client can write to insert-only ACID table	No	Yes	insert_only=true	CREATE TABLE fails