Impala auto-scaling overview

Impala Virtual Warehouses use the following running processes to support auto-scaling:

  • Coordinator processes: Handles all incoming queries, creating execution plans and handing off the query to executor processes for execution. If one coordinator process fails, a backup coordinator process takes over so there is no single point of failure.
  • Executor processes: Processes that execute query fragments. Executor processes run on executor nodes, the unit of sizing for Virtual Warehouses. Each executor node runs one executor:

  • Executor groups: A group of executors that can execute queries. Each executor group can run up to twelve queries. The size of executor groups is determined by the size you choose when you first create the Virtual Warehouse (XSMALL, SMALL, MEDIUM, or LARGE).

    It is recommended that you select the size of a Virtual Warehouse based on the number of nodes you need to run a typical query for your workloads based on average query size and complexity. For example, if you expect that 20 nodes are needed to run a typical query in your workloads, you would create a medium-sized Virtual Warehouse, which by default has 20 executor nodes. However, if you have a large number of queries that must be run in your workloads concurrently, you can set the minimum node count to 40 and maximum node count to 80 by using the Nodes: Min: Max: setting. Then each executor group for this Virtual Warehouse would still contain the 20 executor nodes required to run an average query, but you would have two executor groups to handle the volume of queries in your workloads. If the total number of nodes auto-scales up to 80, then there will be 4 executor groups containing 20 nodes each.

  • Auto-scaler: A process that monitors Impala to determine when more or less resources are needed. When the auto-scaler detects an imbalance in resources, it sends a request to the Kubernetes framework to increase or decrease the number of executor groups in the Virtual Warehouse.

Auto-scaling process

Impala uses memory-based auto-scaling to manage resources:

  1. When users connect to Impala Virtual Warehouses using SQL applications such as Hue, the Impala shell, or other SQL clients that use JDBC or ODBC, the query is handled by the coordinator process. First, the coordinator generates an execution plan for the query.
  2. Then the coordinator locates an executor group that has enough available memory to run the query. Each executor group is limited by the number of queries and the memory available to run them. Currently, executor groups can handle up to 12 queries. If there are no available executor groups to handle a query, it is queued until resources become available.
  3. When the auto-scaler detects that there are queued queries, it adds executor groups to the Virtual Warehouse to execute the queries. The auto-scaler starts scaling up based on what has been set for the auto-scaling mode: Conservative, Balanced, or Aggressive. The auto-scaler scales down the number of executor groups when they are idle according to the auto-scale mode setting as well.

When the auto-scaler has scaled back to the last executor group, which contains the default number of executors for the Virtual Warehouse, and those executors are idle, the Virtual Warehouse is suspended. The time it takes to suspend the Virtual Warehouse is determined by the value that is set for AutoSuspend Timeout. The default number of executors per executor group is based on the number of executor nodes contained by the original size of the Virtual Warehouse when it was created. For example, if the Virtual Warehouse was created as MEDIUM-sized, which has 20 executor nodes, then each executor group contains 20 executors.

If a query is executed on a suspended Virtual Warehouse, then the coordinator queues the query. When the auto-scaler detects a queued query, it immediately adds an executor group that can run the query on the Virtual Warehouse.