Cloudera Docs

Enabling Resource Management with Control Groups

Enabling Linux Control Groups (cgroups) using Cloudera Manager.

You can configure Cloudera Manager to use Linux Control Groups (cgroups) to manage cluster resources. After enabling cgroups for Resource management, you use Service configuration properties to allocate resources by CPU shares, I/O, and memory.

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

Cgroups-based resource management can be enabled for all hosts, or on a per-host basis.
  1. Click Hosts > Host Configuration.
  2. Click Category > Resource Management.
  3. Select the Enable Cgroup-based Resource Management parameter.
  4. To enable Cgroups only on specific hosts:
    1. Click the Add Host Overrides link.
      The Add Host Overrides page displays.
    2. Select the hosts where you want to enable Cgroups.
    3. Click the Add button.
      The Host Configuration page displays.
    4. De-select the All Hosts option in the Enable Cgroup-based Resource Management configuration.
    5. Click Save Changes.
  5. Restart all roles on the host(s).
  6. To configure the default Cloudera Manager resource parameters, see Configuring Resource Parameters. If you are using Custom Cgroups to allocate resources, you configure those resource parameters in the Linux environment.
Limitations
  • Role group and role instance override cgroup-based resource management parameters must be saved one at a time. Otherwise some of the changes that should be reflected dynamically will be ignored.
  • The role group abstraction is an imperfect fit for resource management parameters, where the goal is often to take a numeric value for a host resource and distribute it amongst running roles. The role group represents a "horizontal" slice: the same role across a set of hosts. However, the cluster is often viewed in terms of "vertical" slices, each being a combination of worker roles (such as TaskTracker, DataNode, RegionServer, Impala Daemon, and so on). Nothing in Cloudera Manager guarantees that these disparate horizontal slices are "aligned" (meaning, that the role assignment is identical across hosts). If they are unaligned, some of the role group values will be incorrect on unaligned hosts. For example a host whose role groups have been configured with memory limits but that's missing a role will probably have unassigned memory.