Cloudera Docs

Static Service Pools

Static service pools is a Cloudera Manager feature that allows you to partition cluster resources by service.

This feature provides a simplified, wizard-driven workflow for configuring the underlying Linux Control Groups (cgroups) on your cluster hosts. By allocating a static percentage of resources—CPU, memory, and I/O—to each service, you can prevent a high-load service from consuming all host resources and negatively impacting other services, a situation often referred to as the noisy neighbor problem.

Resource Management is implemented at the role group level within a cluster. When you complete the wizard, Cloudera Manager automatically calculates and applies the appropriate cgroup parameters and application-level memory configurations based on the percentages you have assigned.

The Static Service Pools wizard accomplishes two primary configuration tasks:
  • Cgroup Parameter Configuration: It translates your percentage-based allocations into specific Linux cgroup parameter values for each service's roles. For example, it sets CPU shares/weights and I/O weights to manage resource contention.
  • Cooperative Memory Management: Instead of relying solely on the cgroup memory soft and hard limits, which can cause the kernel to terminate processes abruptly, Static Service Pools also set cooperative memory limits. This is primarily achieved by configuring the maximum Java heap size for Java-based services. This ensures that applications are aware of their memory boundaries and can handle memory allocation failures more gracefully, providing a better user experience than having processes unexpectedly killed by the kernel's Out-Of-Memory (OOM) killer.

It is important to note that the behavior of resource control, particularly for I/O, differs between cgroup v1 and cgroup v2. Cgroup v1 I/O weighting primarily affects read requests, whereas cgroup v2 provides more comprehensive control over both read and write I/O. The Static Service Pool wizard presents distinct configuration tables for cgroup v1 and cgroup v2 to account for these differences.

Viewing Static Service Pool Status

Select Clusters > Cluster name > Static Service Pools. If the cluster has a YARN service, the Static Service Pools Status tab displays and shows whether resource management is enabled for the cluster and the currently configured service pools.

Frequently Asked Questions

Which worker roles get configured by Static Service Pools?
The following worker roles are configured by the Static Service Pools wizard for each service:

  • Impala Service: Impala Daemon role

  • Solr Service: Solr Server role

  • YARN Service: Nodemanager role

  • HDFS Service: Datanode role

  • HBase Service: RegionServer role

  • MapReduce Service: TaskTracker role

Are Linux Control Groups (cgroups) used for managing memory limits?

No, memory management is controlled through Java or service-specific configurations.

Why can’t I use Linux Control Groups (cgroups) soft and hard limits for managing memory?

When a service exceeds a cgroup hard limit, the Linux kernel might abruptly terminate the entire process to enforce the limit. This is a harsh action that can cause unexpected service outages and is handled by the kernel's Out-Of-Memory (OOM) killer. While memory soft limits are less aggressive than hard limits, they are avoided for the same fundamental risk that the Linux kernel will unexpectedly terminate a service process.

How do the weights assigned to each service for CPU and I/O in Linux Control Groups compare to the default weights assigned to other processes running on the system?

For CGroups V2, the default weight assigned to a process is 100. Therefore, Cloudera processes assigned a weight higher than 100 will gain a greater share of CPU cycles or I/O bandwidth than other system processes. For CGroups V1, the default is 1024, but the same concept of relative proportions applies to the number of resources allocated to a process.

Why do I see a value of -1 for a Linux Control Group (cgroup) configuration?

This is the default Cloudera Manager value put in place with Linux Control Groups are not enabled.

Does using the Static Service Pools wizard affect the amount of memory allocated to containers created by YARN?

No, the memory settings configured by the Static Service Pools wizard only apply to the NodeManager role's process. The amount of memory available for YARN containers is determined by the yarn.nodemanager.resource.memory-mb property, which is assigned a default value by Cloudera Manager.

Can I use CGroup configurations to limit how much memory YARN containers can use?

You should use the memory setting yarn.nodemanager.resource.memory-mb to set how much memory YARN can allocate to containers it creates. The setting yarn.nodemanager.resource.memory.cgroups.soft-limit-percentage might be confused with such control, but is actually used to throttle memory utilization within a container once it reaches the set limit, which has a default value of 90%.