Using Apache Hive
Also available as:
PDF

HMS table storage

You need to understand how Hive metastore (HMS) stores Hive tables when you run a CREATE TABLE statement or migrate a table to HDP during an upgrade from an HDP 2.6 cluster. The success or failure of the statement, the resulting table type, and the table location depends on a number of factors.

HMS table transformations

The HMS includes the following Hive metadata about tables that you create:
  • A table definition
  • Column names
  • Data types
  • Comments in a central schema repository
When you use EXTERNAL keyword in the CREATE TABLE statement, HMS stores the table as an external table. When you omit the EXTERNAL keyword and create a managed table, or ingest a managed table, HMS might translate the table into an external table or the table creation can fail, depending on the table properties. An important table property that affects table transformations is the ACID or Non-ACID table type:
Non-ACID
Table properties do not contain any ACID related properties set to true. For example, the table does not contain such properties transactional=true or insert_only=true.
ACID
Table properties do contain one or more ACID properties set to true.
Full ACID
Table properties contain transactional=true but not insert_only=true.
Insert-only ACID
Table properties contain insert_only=true.
The following matrix shows the table type and whether or not the location property is supported.
ACID Managed Location Property Comments Action
Non-ACID Yes Yes Migrated from an HDP 2.6 cluster Table stored as external
Non-ACID, ACID, full ACID, insert-only ACID Yes No Table location is null Table stored in subdirectory metastore.warehouse.external.dir

HMS detects type of client for interacting with HMS, for example Hive or Spark, and compares the capabilities of the client with the table requirement. HMS performs the following actions, depending on the results of the comparison:

Table requirement Client meets requirements Managed Table ACID table type Action
Client can write to any type of ACID table No Yes Yes CREATE TABLE fails
Client can write to full ACID table No Yes insert_only=true CREATE TABLE fails
Client can write to insert-only ACID table No Yes insert_only=true CREATE TABLE fails