Hive VW isolation auto-scaling
The Hive Virtual Warehouse in Cloudera Data Warehouse
on cloud bases its auto-scaling on the size of data scanned for a
query. Queries that scan amounts of data that exceed the
hive.query.isolation.scan.size.threshold setting use the query isolation
method of auto-scaling.
To run ETL-type queries, you turn on query isolation when you configure the Hive Virtual
Warehouse. Scaling occurs as follows:
- If the query does not exceed the hive.query.isolation.scan.size.threshold, concurrency scaling occurs.
- If the query does exceed the hive.query.isolation.scan.size.threshold, query isolation scaling occurs.
HiveServer generates a preliminary query execution plan.
The Hive Virtual Warehouse adds a new dedicated executor group with the right number of
executors to handle the query instead of using the number associated with the T-shirt size you
configured. This isolated executor group is used only under the following conditions:
- You run a single ETL-type query.
- The query exceeds the hive.query.isolation.scan.size.threshold.
Otherwise, the concurrency method of auto-scaling is used.
The following diagram shows the isolation method of auto-scaling:
The variable-sized, ephemeral executor groups for ETL terminate after the query completes.
