Monitoring Resource Management

Cloudera Manager 4 introduced the ability to partition resources across HBase, HDFS, Impala, MapReduce, and YARN services by allowing you to set configuration properties that were enforced by Linux control groups (Linux cgroups). With Cloudera Manager 5, the ability to statically allocate resources using cgroups is configurable through a single static service pool wizard. You allocate services a percentage of total resources and the wizard configures the cgroups.

Monitoring Static Service Pools

Static service pools isolate the services in your cluster from one another, so that load on one service has a bounded impact on other services. Services are allocated a static percentage of total resources—CPU, memory, and I/O weight—which are not shared with other services. When you configure static service pools, Cloudera Manager computes recommended memory, CPU, and I/O configurations for the worker roles of the services that correspond to the percentage assigned to each service. Static service pools are implemented per role group within a cluster, using Linux control groups (cgroups) and cooperative memory limits (for example, Java maximum heap sizes). Static service pools can be used to control access to resources by HBase, HDFS, Impala, MapReduce, Solr, Spark, YARN, and add-on services. Static service pools are not enabled by default.

Viewing Static Service Pools

Select Clusters > Cluster name > Resource Management > Static Service Pools.If the cluster has a YARN service, the Static Service Pools Status tab displays and shows whether resource management is enabled for the cluster, and the currently configured service pools.

Static Service Pool Status

The Status tab of the Static Service Pools page contains a list of current services that can or have been allocated resources and a set of resource usage charts for the cluster.

Click Historical Data to display detailed resource usage charts for each service.

Click a duration link at the top right of the charts to change the time period for which the resource usage displays.

Monitoring Dynamic Resource Pools

A dynamic resource pool is a named configuration of resources and a policy for scheduling the resources among YARN applications and Impala queries running in the pool. Dynamic resource pools allow you to schedule and allocate resources to YARN applications and Impala queries based on a user's access to specific pools and the resources available to those pools. If a pool's allocation is not in use it can be given to other pools. Otherwise, a pool receives a share of resources in accordance with the pool's weight. Dynamic resource pools have ACLs that restrict who can submit work to and administer them.

Viewing Dynamic Resource Pools

  1. Select Clusters > Cluster name > Resource Management > Dynamic Resource Pools. If the cluster has a YARN service, the Dynamic Resource Pools Status tab displays. If the cluster has only an Impala service enabled for dynamic resource pools, the Dynamic Resource Pools Configuration tab displays.

Dynamic Resource Pools Status

For the YARN Independent RM and YARN and Impala Integrated RM resource management scenarios described in Managing Resources with Cloudera Manager, the Dynamic Resource Pools Status tab displays a summary of YARN scheduler status, a list of current allocations for the pools that can or have been allocated resources, and a set of pool resource usage charts for the cluster. If the Impala Independent RM scenario is in effect, there is no Status tab.

Click a duration link at the top right of the charts to change the time period for which the resource usage displays.

  • Status - a summary of the virtual CPU cores and memory that can be allocated by the YARN scheduler.
  • Pools Status - a list of pools that have been explicitly configured and pools created by YARN, properties of the pools, and an action menu.
    • Allocated Memory - The memory assigned to the pool that is currently allocated to applications and queries.
    • Allocated VCores - The number of virtual CPU cores assigned to the pool that are currently allocated to applications and queries.
    • Allocated Containers - The number of YARN containers assigned to the pool whose resources have been allocated.
    • Pending Containers - The number of YARN containers assigned to the pool whose resources are pending.
    • Click and select
      • YARN Applications to display the YARN Applications page and list the applications that are running or have run in that pool.
      • Impala Queries to display the Monitoring Impala Queries page and list the queries that are running or have run in that pool.