Quota management in Cloudera Data Warehouse on premises

How quota management works in Cloudera Data Warehouse

A namespace is created when you create a Database Catalog, a Cloudera Data Visualization instance, or a Virtual Warehouse. At this stage, a resource pool is created with the necessary resources under the selected parent resource pool. You can view the resource pools on the Resource Utilization page by going to the Cloudera Management Console > Resource Utilization > Quotas tab. Virtual Warehouse namespaces allocate a minimum resource quota, known as the Guaranteed Quota in the UI, to ensure the operation of at least one executor group.

If you have configured auto-scaling of the executor groups for a Hive or an Impala Virtual Warehouse and the Virtual Warehouse needs to schedule more executors for the current load, Cloudera Data Warehouse tries to increase its resource pool’s quota. Auto-scaling fails if the requested resource pool quota exceeds the parent resource pool’s quota.

If you have configured auto-scaling of the executor groups for a Hive or an Impala Virtual Warehouse, the requested quota is the resources that are used by the maximum number of executors groups. During a scale-out event, new executor groups are created, and Yunikorn schedules them based on the availability of resources at that moment.

On activating an environment, a root.<environment-name>.cdw resource pool is created by default. The created resource pool is a top-level resource pool that can be assigned to any of the Cloudera Data Warehouse entities that are created under the associated environment. The newly created root.<environment-name>.cdw resource pool has infinite quotas set that can be modified by an administrator from Cloudera Management Console > Resource Utilzation > Quotas.

During environment activation, a monitoring namespace and a separate resource pool are also created under the root.<environment-name>.cdw resource pool for the Diagnostic Data Generation jobs regardless of whether Quota Management is enabled or not. This resource pool is a leaf level resource pool associated with the monitoring namespace.

If the Store Logs on HDFC advanced configuration is enabled, the created log router namespace is enrolled into Quota Management based on the state of Quota Management in Advanced Configurations.

What happens when the quota is insufficient

Suppose you are creating a new Cloudera Data Warehouse entity, such as a Virtual Warehouse with a specific resource pool. The Virtual Warehouse creation process fails if there are insufficient resources when a namespace is being created.

Cloudera CLI support

By using the --resource-pool Cloudera CLI option of the type string, you can specify the resource pool for the Cloudera Data Warehouse entities, namely the Virtual Warehouse and the Cloudera Data Visualization instance, when using Cloudera CLI commands.