Quota management in CDW Private Cloud

Review how to enable quota management in Cloudera Data Warehouse (CDW) on Private Cloud and assign quota-managed resource pools to environments, Data Catalogs, Virtual Warehouses, and Data Visualization instances.

How quota management works in CDW

A namespace is created when you create a Database Catalog or a Virtual Warehouse. At this stage, a resource pool is created with the necessary resources under the selected parent resource pool. You can view the resource pools on the Resource Utilization page by going to the Management Console > Resource Utilization > Quotas tab. Virtual Warehouse namespaces always reserve enough resources to run at least one executor group.

If you have configured auto-scaling of the executor groups for a Hive or an Impala Virtual Warehouse and the Virtual Warehouse needs to schedule more executors for the current load, CDW tries to increase its resource pool’s quota. Auto-scaling fails if the requested resource pool quota exceeds the parent resource pool’s quota.

When a Virtual Warehouse scales down, CDW lowers the quota on the Virtual Warehouse’s resource pool after downscaling. The Virtual Warehouses, even in the suspended state, will have enough resources to contain one executor group.

The default Database Catalog inherits the resource pool from the environment.

What happens when the quota is insufficient

Suppose you are creating a new CDW entity, such as a Virtual Warehouse with a specific resource pool. The Virtual Warehouse creation process fails if there are insufficient resources when a namespace is being created. However, if auto-scaling is enabled, then CDW can provision extra worker pods and the process completes successfully.

Suppose you are moving an existing non-quota-managed CDW entity under quota management. In that case, the operation succeeds only if there is sufficient CPU and memory for the common pods, a minimum of executors for a workload, and the CDW entity fits into the resource pool.

CDP CLI support

By using the --resource-pool CDP CLI option of the type string, you can specify the resource pool for the CDW entities, namely the Virtual Warehouse, cluster/environment, and the Data Visualization instance, when using CDP CLI commands.