Cloudera Docs

Automating metadata invalidation after compaction

As you insert or delete data from Hive ACID tables, Hive generates delta files. The compaction process consolidates the delta files and keeps the system healthy. If you query Hive ACID tables from an Impala Virtual Warehouse, you need to learn how to automatically invalidate metadata after compaction to prevent a possible query failure.

After compaction of ACID tables, metadata in Impala coordinator local caches might be stale. You might experience a failure (FileNotFoundException) when you query these tables. Invalidation refreshes the metadata and prevents the problem. You can configure automatic metadata invalidation as described in the following procedure.
You must upgrade your Database Catalog to the compatible version for DWX-1.1.2-b2008 or later to use this feature.
  1. In Cloudera Data Warehouse, click Overview.
  2. In an Impala Virtual Warehouse, click Options Edit.
  3. Click Configurations > Impala coordinator.
  4. In Configuration files, select flagfile.
  5. Set auto_check_compaction to true, and click Apply.