Workload Aware Auto-Scaling in Impala (Preview)
Workload Aware Auto-Scaling (WAAS) is available as a technical preview. WAAS allocates Impala Virtual Warehouse resources based on the workload that is running.
The Workload Aware Auto-Scaling (WAAS) technical preview improves auto-scaling for the following types of workloads:
- Mixed or unknown workloads
- On-premises workloads you are migrating to CDW
- Workloads running on a big cluster
- Workloads subject to queries with varying resource requirements
- Workloads you cannot easily segregate into different Virtual Warehouses
- Workloads clients must access from a single static endpoint
If you have already successfully segregated your workloads using multiple Virtual Warehouses, the benefits of enabling WAAS may be limited. The multiple Virtual Warehouse approach typically works well in this case.
Without WAAS, if you have mixed workloads, you size your Virtual Warehouse for the most resource-intensive queries, so the Virtual Warehouse can handle all your workloads.
With WAAS, if you have mixed workloads, you define constraints for the overall warehouse size. You can add multiple sizes of executor group sets to spin up if needed by the workload. Within the executor group set, executor groups spin up or shut down as warranted by the workload. Each executor group set is mapped to an admission control resource pool, discussed later.
An executor group set is analogous to a range of nodes assigned to a Virtual Warehouse of a particular size. Dynamic provisioning of mixed workloads uses two, or more, executor group sets instead of a one-size-fits-all executor group.
- group-set-small configured for 2 executor groups (small)
- group-set-large configured for 8 executor groups (large)
Within any executor group set are executor groups, all of the same size.
You can expect the Small executor group set to handle the majority of cases, similar to a small (tee-shirt sized) Virtual Warehouse. This group keeps minimum resources for services. The Large executor group set can scale up greatly to service analytic queries. In Min No. of Groups, you configure a minimum number of groups to be allocated for processing queries before Impala auto-suspends. You configure a maximum number of groups based on your budget.
When migrating from a bare metal installation where your workload would run on hundreds of nodes, or more, consider also configuring an intermediate size using a Custom Set. You can add up to three Custom Sets for a total of five executor group sets. Cloudera recommends starting with no custom sets, establishing benchmarks, and adding one custom set at a time.
|Without WAAS||With WAAS|
|Executors provisioned when query runs for one-size-fits-all Virtual Warehouse||Executor group sets provisioned dynamically within range limits|
|Scaling occurs in multiples of the one-size-fits-all Virtual Warehouse||Scaling occurs within configured limits of two or more executor group sets|
|Small queries can run on the larger than necessary Virtual Warehouse||Small queries run on a smaller executor group set|
|Unused CPU expenses hard to control||Unused CPU expenses of manageable|
|Large queries cannot run on a small Virtual Warehouse||Large queries run on an executor group of the appropriate size|
|SLA hard to manage||SLA manageable|
Workload Aware Auto-Scaling uses replanning, described in the next topic, to select an executor group set.