This is the documentation for Cloudera Manager 5.1.x. Documentation for other versions is available at Cloudera Documentation.

Managing Resources

Resource management helps ensure predicable behavior by defining the impact of different services on cluster resources. The goals of Cloudera Manager's resource management features are to:
  • Guarantee completion in a reasonable time frame for critical workloads
  • Support reasonable cluster scheduling between groups of users based on fair allocation of resources per group
  • Prevent users from depriving other users access to the cluster

Cloudera Manager provides several features to assist you with allocating cluster resources to services. Initially Cloudera Manager introduced the ability to partition resources across HBase, HDFS, Impala, MapReduce, and YARN services by setting configuration properties that were enforced by Linux control groups (Linux cgroups). With Cloudera Manager 5, the ability to statically allocate resources using cgroups is configurable through a single static service pool user interface.

In addition, Cloudera Manager allows you to manage mechanisms for dynamically apportioning the resources statically allocated to YARN and Impala using dynamic resource pools. Depending on the version of CDH you are using, dynamic resource pools in Cloudera Manager support the following resource management (RM) scenarios:
  • (CDH 5) YARN Independent RM - YARN manages the virtual cores, memory, running applications, and scheduling policy for each pool.
  • (CDH 5) YARN and Impala Integrated RM - YARN manages memory for pools running Impala queries; Impala limits the number of running and queued queries in each pool.
  • (CDH 5) YARN and Impala Independent RM - YARN manages the virtual cores, memory, running applications, and scheduling policy for each pool; Impala manages memory for pools running queries and limits the number of running and queued queries in each pool.
  • (CDH 5 and CDH 4) Impala Independent RM - Impala manages memory for pools running queries and limits the number of running and queued queries in each pool.

The scenario where YARN and Impala resources are managed by YARN requires the Impala Llama ApplicationMaster role. The scenarios where Impala independently manages resources employ Impala admission control. For further information on Impala admission control, see Admission Control and Query Queueing.

If you are running CDH 4 and have a MapReduce service, you can specify how MapReduce jobs share resource by configuring the MapReduce scheduler. See Configuring the MapReduce Scheduler.

Continue reading:

Page generated September 3, 2015.