Behavioral Changes in YARN

Functional adjustments and behavioral updates for YARN and YARN Queue Manager are introduced in Cloudera Runtime 7.3.2, its service packs, and cumulative hotfixes.

Cloudera Runtime 7.3.2.10000 SP1

Summary: Truststore password handling changed for FIPS 140-3 environments

In FIPS 140-3 mode, the default JVM truststore uses BCFKS format, which requires a password to be provided – unlike JKS, which assumes a default password. To preserve the auto-TLS experience and prevent MapReduce task failures, YARN now automatically injects the DEFAULT_TRUSTSTORE_PWD environment variable into containers when a FIPS environment is detected. The value of this variable is passed as the javax.net.ssl.trustStorePassword JVM property at task startup.

No manual configuration of /etc/default/cloudera-scm-server or truststore password settings are required. The default password value (changeit) is used and thus behavior is configurable using Cloudera Manager.

Summary: UpscaleServiceV2 for YARN autoscaling

Previous behaviour – When numerous Application Masters (AMs) are active, the previous (v1) autoscaling mechanism encounters a key limitation. Because it evaluates pending requests strictly against total available cluster capacity, a queue requiring 25% additional resources will fail to trigger scaling if 50% free capacity exists in other parts of the cluster. Consequently, the affected queue becomes frozen due to unfulfilled resource requests.

New behaviour – A new component, UpscaleServiceV2, is now introduced. It does not rely only on available cluster resources and it accounts for pending applications and AM constraints. Because it focuses on pending resource requests without regard to existing capacity, queue configurations do not impact scaling determinations. UpscaleServiceV2 is capable of initiating scale-out operations even when unallocated resources remain available in the cluster. The following new configuration properties are now added and can be configured through the Cloudera Manager UI:

Table 1. New configuration properties
Property	Default	Description
`yarn.cluster.scaling.upscale.version`	1	Set to `2` to enable UpscaleServiceV2
`yarn.cluster.scaling.upscale.am.multiplier`	2.0	Buffer multiplier for AM capacity headroom
`yarn.cluster.scaling.upscale.threshold.cpu`	0	Minimum pending CPU before scaling triggers
`yarn.cluster.scaling.upscale.threshold.memory`	0	Minimum pending memory before scaling triggers
`yarn.cluster.scaling.upscale.threshold.apps`	0	Minimum pending applications before scaling triggers

Cloudera Runtime 7.3.2.100 CHF 1

There are no behavioral changes in this release.

Cloudera Runtime 7.3.2

There are no behavioral changes introduced in this release.