Auto-scaling Virtual Warehouses
This topic provides an overview of auto-scaling in Cloudera Data Warehouse (CDW) Private Cloud.
Virtual Warehouses can use Hive or Impala as the underlying execution engine. Typically, Hive is used to support complex reports and enterprise dashboards. Impala is used to support interactive, ad-hoc analysis. When you create a Virtual Warehouse, you set auto-scaling to make sure you have adequate resources to meet increases in demand. Auto-scaling settings also insure that your Virtual Warehouse relinquishes resources when demand decreases to save costs.
Auto-scaling: where scaling and concurrency meet
Scaling is the total capacity of the system and how elastic it is. System capacity requirements are based on the size of the largest query you need to run on a warehouse. Concurrency is the number of queries that can run at the same time in the same Virtual Warehouse.
In traditional deployments, scaling and concurrency must be planned for before you deploy your warehouse. In the cloud, the ability to acquire better scaling and concurrency elastically in response to workload demand enables the system to operate more efficiently than the maximum limits you plan for. If you run your Virtual Warehouse configured to accommodate your peak workload as a constant default configuration, you might have inefficient resource utilization when system demand falls below that level.
Caching and auto-scaling
In CDW Private Cloud service frequently accessed data is cached on HDFS. This caching ensures that the data can be quickly retrieved for subsequent queries, boosting data warehouse performance.
Fault-tolerance and auto-scaling
Virtual Warehouses can tolerate single-node failures of any of its workers and can continue running active queries. Auto-scaling separates nodes from each other. This node separation provides better protection when a rogue query is submitted to the warehouse. In this scenario, node failures are limited to the auto-scaled nodes, which limits the impact of the rogue query to a small part of the Virtual Warehouse. The choice to have more auto-scaling groups indirectly allows for the system to tolerate such scenarios, so it is always recommended to use auto-scaling, even if the workload is a predictable one.
