Monitoring Resource Management

With Cloudera Manager 5, statically allocating resources using cgroups is configurable through a single static service pool wizard. You allocate services as a percentage of total resources, and the wizard configures the cgroups.

Monitoring Static Service Pools

Static service pools isolate the services in your cluster from one another, so that load on one service has a bounded impact on other services. Services are allocated a static percentage of total resources—CPU, memory, and I/O weight—which are not shared with other services. When you configure static service pools, Cloudera Manager computes recommended memory, CPU, and I/O configurations for the worker roles of the services that correspond to the percentage assigned to each service. Static service pools are implemented per role group within a cluster, using Linux control groups (cgroups) and cooperative memory limits (for example, Java maximum heap sizes). Static service pools can be used to control access to resources by HBase, HDFS, Impala, MapReduce, Solr, Spark, YARN, and add-on services. Static service pools are not enabled by default.

Viewing Static Service Pools

Select Clusters > Cluster name > Resource Management > Static Service Pools.If the cluster has a YARN service, the Static Service Pools Status tab displays and shows whether resource management is enabled for the cluster, and the currently configured service pools.

Static Service Pool Status

The Status tab of the Static Service Pools page contains a list of current services that can or have been allocated resources and a set of resource usage charts for the cluster.

Click Historical Data to display detailed resource usage charts for each service.

Click a duration link at the top right of the charts to change the time period for which the resource usage displays.

Monitoring Dynamic Resource Pools

A dynamic resource pool is a named configuration of resources and a policy for scheduling the resources among YARN applications and Impala queries running in the pool. Dynamic resource pools allow you to schedule and allocate resources to YARN applications and Impala queries based on a user's access to specific pools and the resources available to those pools. If a pool's allocation is not in use, it can be preempted and distributed to other pools. Otherwise, a pool receives a share of resources in accordance with the pool's weight. Dynamic resource pools have access control lists (ACLs) that restrict who can submit work to and administer them.

Viewing Dynamic Resource Pools

  1. Go to the YARN service.
  2. Click the Resource Pools tab.

Click a duration link at the top right of the charts to change the time period for which the resource usage displays.

  • Status - a summary of the virtual CPU cores and memory that can be allocated by the YARN scheduler.
  • Resource Pools Usage - a list of pools that have been explicitly configured and pools created by YARN, and properties of the pools. The Configuration link takes you to the Dynamic Resource Pool Configuration page.
    • Allocated Memory - The memory assigned to the pool that is currently allocated to applications and queries.
    • Allocated VCores - The number of virtual CPU cores assigned to the pool that are currently allocated to applications and queries.
    • Allocated Containers - The number of YARN containers assigned to the pool whose resources have been allocated.
    • Pending Containers - The number of YARN containers assigned to the pool whose resources are pending.