Cloudera Docs

Managing cluster resources using Quota Management (Technical Preview)

You can use Quota Management page in the Management Console to control how cluster resources are allocated.

Overview

Quota management enables you to control how resources are allocated within your CDP Private Cloud Data Services clusters. In order to prevent a single job or service from consuming all available cluster resources, you can limit the number of CPUs and memory allocated by application, user, business units, or Data Service by defining resource pools that define resource limits. Pools are organized in a hierarchical manner by defining nodes in the hierarchy with resource limits, which can then be subdivided as needed to allocate resources for an organization and to allocate resources to cluster-wide services such as the Control Plane. You can also use this feature to audit job ownership and resource usage.

Quota management is currently disabled by default. When enabled, it is only available for use with the Cloudera Data Engineering (CDE) service. Instructions to enable the feature appear later in this document.

A system administrator defines the initial groupings, starting with the default tenant node and first creating child nodes that define quotas for CDP system resources. For this Technical Preview, you will need to edit the default node with the total current resources of your cluster.

Next the administrator creates one or more child nodes, in this example, nodes are created for the marketing and sales business units and a node is created for CDP resources.

When Data Service users create new artifacts in their service’s interface they can reference a pool name in which to run their jobs. The resources available for the job are limited by the quotas established for the pool. For example, the Parent pool, root.default.marketing refers to the “marketing” business unit pool node defined in the hierarchy. The example below shows how to specify a Resource pool when enabling the CDE service.

When you click on the drop-down list for Resource Pool, a list of available pools, as defined on the Quotas page, displays.

Enabling the Quota Management Feature

  1. Open the CDP console.

  2. Change the URL in the browser by adding flags at the end of the URL.

    The Feature Flags page opens.

  3. Slide the toggle switch for Enable Quota Manager to the on position.

Accessing the Quotas Page

  1. Open the CDP console.

  2. Go to the Management Console service.

  3. Click Resource Utilization in the left navigation panel.

  4. Select the Quotas tab.

Creating child nodes (resource pools),

  1. Click the actions menu in a pool and select Add Child.

    The Add Quota dialog box displays.

  2. Enter the following information:
    • Name – Enter a name for this resource pool.
    • Memory – Use the sliders to select the memory allocation for the pool. Use the drop-down list to select the units.
    • CPU – Use the sliders to select the CPU allocation for the pool. You can choose how CPUs are counted by selecting Cores or Millicores from the drop-down list.
    • GPU – Use the sliders to select the GPU allocation for the pool.
    • Validity – Enter the length of time that the pool can remain active. To keep the pool active indefinitely, enter -1.
    • Tags – Tags provide a way to add user-defined name/value pairs as metadata for the pools. This is not currently used in this technical preview.

Note that the maximum value of the sliders is limited by the resources of the parent pool and the amount of resources not used by other pools.

Editing a resource pool

You can edit a resource pool after creating it by clicking the actions menu for the pool and selecting Edit.

Viewing details of a resource pool

You can view the resource allocations and other information by clicking the “i” icon. The namespace for the pool appears at the top:

Sorting the resource pool display.

Click the Sort by drop down list and select a sorting option to sort the resource pools within each level.

Using Resource pools in a Data Service

When creating artifacts in a Data Service, you specify which pool to use to allocate resources to the artifacts.

For information about specifying pools in the Cloudera Data Engineering (CDE) service, see Adding a Cloudera Data Engineering service and Creating virtual clusters .