Configuring Impala VW auto-scaling

Configuring the Impala Virtual Warehouse to use concurrency auto-scaling is critical for controlling cloud expenses.

  • You are familiar with the auto-scaling process.
  • You are creating an Virtual Warehouse for running BI-type queries.
  • In Cloudera Data Warehouse, you added an Impala Virtual Warehouse, configured the size of the Impala Virtual Warehouse, and configured auto-suspend as described in previous topics.
  • You obtained the DWAdmin role.
In this task, you configure the following properties:
  • Scale Up Delay: Sets the length of time in seconds to wait before adding more executors if queries wait in the queue.
  • Scale Down Delay: Sets the length of time in seconds to wait before removing executors if executor groups are idle.

The time to auto-scale up or down is affected by the underlying Kubernetes configuration.

By default Impala Virtual Warehouses can run 3 large queries per executor group. Executors can handle more queries that are simpler and that do not utilize concurrency on the executor.

If you need to tune your Impala Virtual Warehouse to run more than 3 queries per executor group, use Legacy Multithreading mode. The Virtual Warehouse can run 12 queries per executor group in legacy multithreading mode. For most read-only queries the default setting of 3 queries per executor group is sufficient. After creating an Impala Virtual Warehouse, you edit it to enable legacy multithreading.

.
  1. In Scale Up Delay, set the seconds to wait before adding more executors if queries wait in the queue.
  2. In Scale Down Delay, set the seconds to wait before removing executors when executor groups are idle.
  3. Click CREATE.
  4. (Optional) To run 12 queries per executor group instead of the default 3, in the Overview page under Virtual Warehouses, select the Impala Virtual Warehouse, click Options , and select Edit.
  5. In Sizing and Scaling, turn on Enable Legacy Multithreading., and click APPLY.