Auto-scaling Virtual Warehouses

Virtual Warehouses can use Hive or Impala as the underlying execution engine. Typically, Hive is used to support complex reports and enterprise dashboards. Impala is used to support interactive, ad-hoc analysis. When you create a Virtual Warehouse, you set auto-scaling to make sure you have adequate resources to meet increases in demand. Auto-scaling settings also insure that your Virtual Warehouse relinquishes resources when demand decreases to save costs.

Auto-scaling: where scaling and concurrency meet

Scaling is the total capacity of the system and how elastic it is. System capacity requirements are based on the size of the largest query you need to run on a warehouse. Concurrency is the number of queries that can run at the same time in the same Virtual Warehouse.

In traditional deployments, scaling and concurrency must be planned for before you deploy your warehouse. In the cloud, the ability to acquire better scaling and concurrency elastically in response to workload demand enables the system to operate at a lower cost than the maximum limits you plan for. If you run your Virtual Warehouse configured to accommodate your peak workload as a constant default configuration, you might have cost overruns when system demand falls below that level.

Caching and auto-scaling

In CDP Data Warehouse service frequently accessed data from the object store is cached on instance storage within each executor node. Examples of the object store are S3, ADLS, and so on. This caching ensures that the data can be quickly retrieved for subsequent queries, boosting data warehouse performance.

Fault-tolerance and auto-scaling

Virtual Warehouses can tolerate single-node failures of any of its workers and can continue running active queries. Auto-scaling separates nodes from each other. This node separation provides better protection when a rogue query is submitted to the warehouse. In this scenario, node failures are limited to the auto-scaled nodes, which limits the impact of the rogue query to a small part of the Virtual Warehouse. The choice to have more auto-scaling groups indirectly allows for the system to tolerate such scenarios, so it is always recommended to use auto-scaling, even if the workload is a predictable one.