Cloudera Docs

Configuring intermediate results caching

Learn about the configurations required to enable the intermediate results cache for Impala queries.

To use the intermediate results cache, you must configure the following settings. By default, these features are disabled.

In Cloudera Data Warehouse, intermediate results cache storage is shared with the data cache and scratch space. To accommodate this usage, you might need to adjust the existing quotas for these elements, such as the --data_cache startup flag.

  1. Login to the Cloudera web interface and navigate to the Cloudera Data Warehouse service.
  2. In the Cloudera Data Warehouse service, go to Virtual Warehouses in the left navigation panel.
  3. In the Impala Virtual Warehouse, select the Details option from the drop-down list.
  4. Go to Details and then to the Configurations tab.
  5. Configure cache for Impala Coordinator.
    1. Go to Impala Coordinator.
    2. Under Configuration files, select the flagfile.
    3. Click Add Custom Configuration.
    4. --allow_tuple_caching=true --tuple_cache=[directory_path]:[capacity]
      Add the flags
      --tuple_cache=/opt/impala/cache:20GB
    5. Click Add.
    6. Search for default_query_options and add the enable_tuple_cache=true value.
  6. Configure cache for Impala Executor.
    1. Go to Impala Executor.
    2. Under Configuration files, select the flagfile.
    3. Click Add Custom Configuration.
    4. --allow_tuple_caching=true --tuple_cache=[directory_path]:[capacity]
      Add the flags
    5. Click Add.
  7. Click Apply Changes and restart Impala.

The Virtual Warehouse now stores intermediate query results in the specified local directory. Subsequent queries with matching plan fragments can retrieve data from the cache, which reduces execution time and resource consumption.

You can monitor cache hits and performance by checking the Impala Query Profile. The profile displays metrics for tuple cache hits under the relevant plan nodes.