INVALIDATE METADATA statement
INVALIDATE METADATA statement marks the metadata for one or all
tables as stale. The next time the Impala service performs a query against a table whose
metadata is invalidated, Impala reloads the associated metadata before the query proceeds. As
this is a very expensive operation compared to the incremental metadata update done by the
REFRESH statement, when possible, prefer
REFRESH rather than
INVALIDATE METADATAis required when the following changes are made outside of Impala, in Hive and other Hive client, such as SparkSQL:
- Metadata of existing tables changes.
- New tables are added, and Impala will use the tables.
DATABASElevel privileges are changed from outside of Impala.
- Block metadata changes, but the files remain the same (HDFS rebalance).
- UDF jars change.
- Some tables are no longer queried, and you want to remove their metadata from the catalog and coordinator caches to reduce memory requirements.
INVALIDATE METADATA is needed when the changes are made by
Once issued, the
INVALIDATE METADATA statement cannot be cancelled.
INVALIDATE METADATA [[db_name.]table_name]
If there is no table specified, the cached metadata for all tables is flushed and synced with Hive Metastore (HMS). If tables were dropped from the HMS, they will be removed from the catalog, and if new tables were added, they will show up in the catalog.
If you specify a table name, only the metadata for that one table is flushed and synced with the HMS.
To return accurate query results, Impala need to keep the metadata current for the
databases and tables queried. Therefore, if some other entity modifies information used by
Impala in the metastore, the information cached by Impala must be updated via
INVALIDATE METADATA or
INVALIDATE METADATAis an asynchronous operations that simply discards the loaded metadata from the catalog and coordinator caches. After that operation, the catalog and all the Impala coordinators only know about the existence of databases and tables and nothing more. Metadata loading for tables is triggered by any subsequent queries.
REFRESHreloads the metadata synchronously.
REFRESHis more lightweight than doing a full metadata load after a table has been invalidated.
REFRESHcannot detect changes in block locations triggered by operations like HDFS balancer, hence causing remote reads during query execution with negative performance implications.
REFRESH after invalidating a specific table to separate the metadata
load from the first query that's run against that table.
This example illustrates creating a new database and new table in Hive, then doing an
INVALIDATE METADATA statement in Impala using the fully qualified table
name, after which both the new table and the new database are visible to Impala.
INVALIDATE METADATA statement was issued, Impala would give a
not found error if you tried to refer to those database or table names.
$ hive hive> CREATE DATABASE new_db_from_hive; hive> CREATE TABLE new_db_from_hive.new_table_from_hive (x INT); hive> quit; $ impala-shell > REFRESH new_db_from_hive.new_table_from_hive; ERROR: AnalysisException: Database does not exist: new_db_from_hive > INVALIDATE METADATA new_db_from_hive.new_table_from_hive; > SHOW DATABASES LIKE 'new*'; +--------------------+ | new_db_from_hive | +--------------------+ > SHOW TABLES IN new_db_from_hive; +---------------------+ | new_table_from_hive | +---------------------+
REFRESH statement for incremental metadata update.
> REFRESH new_table_from_hive;
By default, the
INVALIDATE METADATA command checks HDFS permissions of
the underlying data files and directories, caching this information so that a statement
can be cancelled immediately if for example the
impala user does not have
permission to write to the data directory for the table. (This checking does not apply
when the catalogd configuration option
--load_catalog_in_background is set to
false, which it
is by default.) Impala reports any lack of write permissions as an
message in the log file.
If you change HDFS permissions to make data readable or writeable by the Impala user,
INVALIDATE METADATA to make Impala aware of the change.
By default, much of the metadata for Kudu tables is handled by the underlying storage layer. Kudu tables have less reliance on the Metastore database, and require less metadata caching on the Impala side. For example, information about partitions in Kudu tables is managed by Kudu, and Impala does not cache any block locality metadata for Kudu tables. If the Kudu service is not integrated with the Hive Metastore, Impala will manage Kudu table metadata in the Hive Metastore.
statements are needed less frequently for Kudu tables than for
HDFS-backed tables. Neither statement is needed when data is
added to, removed, or updated in a Kudu table, even if the changes
are made directly to Kudu through a client program using the Kudu API.
REFRESH table_name or
INVALIDATE METADATA table_name
for a Kudu table only after making a change to the Kudu table schema,
such as adding or dropping a column.