Quota management in Cloudera Data Warehouse on premises

How quota management works in Cloudera Data Warehouse

A namespace is created when you create a Database Catalog, a Cloudera Data Visualization instance, or a Virtual Warehouse. At this stage, a resource pool is created with the necessary resources under the selected parent resource pool. You can view the resource pools on the Resource Utilization page by going to the Cloudera Management Console > Resource Utilization > Quotas tab. Virtual Warehouse namespaces always reserve enough resources to run at least one executor group.

If you have configured auto-scaling of the executor groups for a Hive or an Impala Virtual Warehouse and the Virtual Warehouse needs to schedule more executors for the current load, Cloudera Data Warehouse tries to increase its resource pool’s quota. Auto-scaling fails if the requested resource pool quota exceeds the parent resource pool’s quota.

When a Virtual Warehouse scales down, Cloudera Data Warehouse lowers the quota on the Virtual Warehouse’s resource pool after downscaling. The Virtual Warehouses, even in the suspended state, will have enough resources to contain one executor group.

If Quota Management is enabled, the Database Catalog inherits the resource pool from the environment on activation. If a Database Catalog does not have a resource pool assigned, the same can be assigned later by editing the Database Catalog.

On activating an environment, a root.<environment-name>.cdw resource pool is created by default regardless of whether Quota Management is enabled or not. The created resource pool is a top-level resource pool that can be assigned to any of the Cloudera Data Warehouse entities that are created under the associated environment. The newly created root.<environment-name>.cdw resource pool has infinite quotas set that can be modified by an administrator from Cloudera Management Console > Resource Utilzation > Quotas.

If the Store Logs on HDFC advanced configuration is enabled, the created log router namespace is enrolled into Quota Management based on the state of Quota Management in Advanced Configurations.

What happens when the quota is insufficient

Suppose you are creating a new Cloudera Data Warehouse entity, such as a Virtual Warehouse with a specific resource pool. The Virtual Warehouse creation process fails if there are insufficient resources when a namespace is being created. However, if auto-scaling is enabled, then Cloudera Data Warehouse can provision extra worker pods and the process completes successfully.

Suppose you are moving an existing non-quota-managed Cloudera Data Warehouse entity under quota management. In that case, the operation succeeds only if there is sufficient CPU and memory for the common pods, a minimum of executors for a workload, and the Cloudera Data Warehouse entity fits into the resource pool.

Cloudera CLI support

By using the --resource-pool Cloudera CLI option of the type string, you can specify the resource pool for the Cloudera Data Warehouse entities, namely the Virtual Warehouse, Database Catalog, and the Cloudera Data Visualization instance, when using Cloudera CLI commands.