Assigning applications and queries to resource pools

You can specify at run time that an application or query must run in a named resource pool.

  • To specify a pool at run time for a YARN application, provide the pool name in the mapreduce.job.queuename property.
  • To specify a pool at run time for an Impala query, provide the pool name in the REQUEST_POOL option.
Cloudera Manager allows you to specify a set of ordered rules for assigning applications and queries to pools. You can also specify default pool settings directly in the YARN fair scheduler configuration.

Some rules allow to you specify that the pool be created in the dynamic resource pool configuration if it does not already exist. Allowing pools to be created is optional. If a rule is satisfied and you do not create a pool, YARN runs the job "ad hoc" in a pool to which resources are not assigned or managed.

If no rule is satisfied when the application or query runs, the YARN application or Impala query is rejected.

Placement Rule Ordeing and Evaluation

Pool placement rules are evaluated in the order in which they appear in the placement rule list. When a job is submitted, the rules are evaluated, and the first matching rule is used to determine the pool in which the job is run.

If a rule is always satisfied, subsequent rules are not evaluated. Rules that are never evaluated appear in struck-though gray text.

By default, pool placement rules are ordered in reverse order of when they were added; the last rule added appears first. You can easily reorder rules.

Default Placement Rules and Order

The default placement rules and order for YARN are:
  1. Use the pool specified at run time and create the pool if it does not exist.
  2. Use the pool root.users.[username].
  3. Use the pool root.default.