Resource Management
Resource management helps ensure predictable behavior by defining the impact of different services on cluster resources.
Use resource management to:
- Guarantee completion in a reasonable time frame for critical workloads.
- Support reasonable cluster scheduling between groups of users based on fair allocation of resources per group.
- Prevent a single user from creating a noisy neighbor situation that deprives other users access to the cluster.
Statically allocating resources using Control Groups (cgroups) is configurable through a single static service pool wizard. You allocate services as a percentage of total resources, and the wizard configures the cgroups.
For example, the following figure illustrates static pools for HBase, HDFS, Impala, and YARN services that are respectively assigned 20%, 30%, 20%, and 30% of cluster resources.
You can dynamically apportion resources that are statically allocated to YARN and Impala by using dynamic resource pools.
Depending on the version of Cloudera Runtime you are using, dynamic
resource pools in Cloudera Manager support the following scenarios:
- YARN - YARN manages the virtual cores, memory, running applications, maximum resources for undeclared children (for parent pools), and scheduling policy for each pool. In the preceding diagram, three dynamic resource pools—Dev, Product, and Mktg with weights 3, 2, and 1 respectively—are defined for YARN. If an application starts and is assigned to the Product pool, and other applications are using the Dev and Mktg pools, the Product resource pool receives 30% x 2/6 (or 10%) of the total cluster resources. If no applications are using the Dev and Mktg pools, the YARN Product pool is allocated 30% of the cluster resources.
- Impala - Impala manages memory for pools running queries and limits the number of running and queued queries in each pool.
