Workload-aware autoscaling tuning
You can use an Impala query option to control query routing. You can also set a parameter to restrict queries from running in parallel under certain circumstances.
With WAAS, Impala internally uses the REQUEST_POOL query option to control which executor group set is used for the query. WAAS is sophisticated, but if a query is being sent to the wrong pool, you can use the REQUEST_POOL query option to override the executor group set to be used.
Request pool query option
You can override Impala using the REQUEST_POOL query option to select a particular executor group set if you want. For example, high query latency might indicate the query is not being routed to the correct pool. You might get better performance from a larger pool than the REQUEST_POOL. Your large queries can be targeted at a specific pool using a request pool.
The following example overrides Impala using the REQUEST_POOL query option to route large queries to group-set-large:
set request_pool="group-set-large"
Parallelism query option
Workload Aware Auto-Scaling includes the capability in the Impala planner to reduce query parallelism under certain conditions.
The MT_DOP multi-threaded planning for queries in Cloudera Data Warehouse was previously static in nature. A query would use 12 threads per node and aggressively use all cores. In some cases, using all cores is unnecessary. A new feature can reduce query parallelism to more effectively use resources. Sophisticated planner functionality makes passes over the plan to determine how many cores are optimal for the query. Intelligence is gathered about fragments, how much work is involved in each fragment, and blocking operations in the plan.
Traversing fragments of the query plan tree, the planner propagates estimates of the max core count. Many details are gathered to determine the optimal parallelism for a pool. If parallelism is greater than the threshold (admission-controller.pool-max-query-cpu-core-per-node-limit), traverse to the next larger pool. The information is analyzed and the planner restricts parallelism for fragments that require few threads. Costs are reduced.
To enable this feature, you restrict parallelism by setting the COMPUTE_PROCESSING_COST=1 query option.
