Metadata Management

In previous versions of Impala, every coordinator kept a replica of all the cache in catalogd, consuming large memory on each coordinator with no option to evict. Metadata always propagated through the statestored and suffers from head-of-line blocking, for example, one user loading a big table blocking another user loading a small table.

With this new feature, the coordinators pull metadata as needed from catalogd and cache it locally. The cached metadata gets evicted automatically under memory pressure.

The granularity of on-demand metadata fetches is now at the partition level between the coordinator and catalogd. Common use cases like add/drop partitions do not trigger unnecessary serialization/deserialization of large metadata.

This feature is disabled by default.

Limitation:

Global INVALIDATES are not supported when this feature is enabled. If your workload requires global INVALIDATES, do not use this feature.

To keep the size of metadata bounded, catalogd periodically scans all the tables and invalidates those not recently used. There are two types of configurations in catalogd.

Time-based cache invalidation

Catalogd invalidates tables that are not recently used in the specified time period (in seconds).

The ‑‑invalidate_tables_timeout_s flag needs to be applied to both impalad and catalogd.

Memory-based cache invalidation

When the memory pressure reaches 60% of JVM heap size after a Java garbage collection in catalogd, Impala invalidates 10% of the least recently used tables.

The ‑‑invalidate_tables_on_memory_pressure flag needs to be applied to both impalad and catalogd.

Automatic invalidation of metadata provides more stability with lower chances of running out of memory, but the feature could potentially cause performance issues and may require tuning.

When tools such as Hive and Spark are used to process the raw data ingested into Hive tables, new HMS metadata (database, tables, partitions) and filesystem metadata (new files in existing partitions/tables) is generated. In previous versions of Impala, in order to pick up this new information, Impala users needed to manually issue an INVALIDATE or REFRESH commands.

When automatic invalidate/refresh of metadata is enabled, catalogd polls Hive Metastore (HMS) notification events at a configurable interval and processes the following changes:

• Invalidates the tables when it receives the ALTER TABLE event.
• Refreshes the partition when it receives the ALTER, ADD, or DROP partitions.
• Adds the tables or databases when it receives the CREATE TABLE or CREATE DATABASE events.
• Removes the tables from catalogd when it receives the DROP TABLE or DROP DATABASE events.
• Refreshes the table and partitions when it receives the INSERT events.

  If the table is not loaded at the time of processing the INSERT event, the event processor does not need to refresh the table and skips it.

This feature is controlled by the ‑‑hms_event_polling_interval_s flag. Start the catalogd with the ‑‑hms_event_polling_interval_s flag set to a positive integer to enable the feature and set the polling frequency in seconds. We recommend the value to be less than 5 seconds.

The following use cases are not supported:

• When you bypass HMS and add or remove data into table by adding files directly on the filesystem, HMS does not generate the INSERT event, and the event processor will not invalidate the corresponding table or refresh the corresponding partition.

  It is recommended that you use the LOAD DATA command to do the data load in such cases, so that event processor can act on the events generated by the LOAD command.

• The Spark API that saves data to a specified location does not generate events in HMS, thus is not supported. For example:

  Seq((1, 2)).toDF("i", "j").write.save("/user/hive/warehouse/spark_etl.db/customers/date=01012019")

This feature is turned off by default with the ‑‑hms_event_polling_interval_s flag set to 0.