Cloudera Manager 7.6.7 Cumulative hotfix 4

OPSAPS-63529

The "deleteLatestSourceSnapshotOnJobFailure" HDFS policy property could be accessed only using CLI. You can now configure this parameter during HDFS replication policy creation using the Advanced > Restart replication using non-incremental (bootstrap) replication on replication failure field.

OPSAPS-63571

Sometimes, entries reported by the HDFS snapshot-diff report for deleted directories appear as modified. This might raise an FileNotFoundException error. In this scenario, you can configure the "com.cloudera.enterprise.distcp.hdfs-snapshot-diff-cleanup.enabled" advanced configuration snippet to address these unexpected entries.

OPSAPS-63930

By default, snapshot diff-based (incremental) HDFS - HDFS replication uses a temp directory, created in the parent of replication destination directory to synchronize source-side rename and delete operations: deleted and renamed paths are first moved into this temporary directory, then the renamed ones will be moved to their target followed by the deletion of this temporary directory (thus deleting the paths scheduled to be deleted). Note that OPSAPS-63759 provides an optional behavior to execute individual deletes without these moves.

This behavior of incremental replication leads to failure and fallback to bootstrap (full file listing) replication when the replication process can not create this temporary directory (due to restrictive HDFS permissions) or when the replication destination contains one or more HDFS encryption zones (because HDFS moves can not cross encryption zone boundary).

This optional workaround solves these problems by executing rename operations in-place when possible, otherwise using the best possible temporary rename operations without the need of the above mentioned common temporary directory. Note that this workaround can be considered as a superset of OPSAPS-63759. That is when both are enabled, the current one is applied.

Activating this workaround:

• Set HDFS service core-site.xml advanced configuration snippet (on the destination side) "com.cloudera.enterprise.distcp.direct-rename-and-delete.enabled" to "true".
• In an incremental replication run, check the stderr log of the last "Trigger a HDFS replication job on one of the available HDFS roles." step, and make sure the INFO distcp.DistCpSync: Will use direct rename and delete (for non cloud target) when using snapshot diff based sync. Temp directory creation on the target will be skipped. message is displayed.

Adjusting delete logging: By default, every 100000 direct delete operations executed by this workaround are logged. This is useful for following the synchronization of large source side deletes. This default interval can be overridden by setting the "com.cloudera.enterprise.distcp.direct-delete.log-interval" advanced configuration snippet to an integer value greater than 0. Note that this advanced configuration snippet is shared with a workaround in OPSAPS-63759.

Usage notes: There can be conflicting source side renames and rename - delete interactions when their destination side replay need to use temporary renames (for example, a name swap between two paths using three renames). For these cases, the temporary rename destination will typically be next to the final rename destination (will share the same parent path) avoiding both above mentioned failure scenarios. Such temporary renames will be logged during execution like:

distcp.DistCpSync: Executing a temp rename: /test-repl-target/test-repl-source/file2 -> /test-repl-target/test-repl-source/file2748016654

After execution, the number of operations will also be logged like:

INFO distcp.DistCpSync: Synced 0 through-tmp/cloud rename(s) and 0 through-tmp delete(s) to target.
INFO distcp.DistCpSync: Synced 2 direct delete(s) to target.
INFO distcp.DistCpSync: Synced 2 direct rename(s) to target.
INFO distcp.DistCpSync: Used 2 additional temporary rename(s) during syncing.

OPSAPS-64925

You could configure the numListstatusThreads parameter, that specifies the number of threads to be used for fetching the file statuses, only through CLI and not during the HDFS replication policy creation process. This issue is fixed.

You can now configure this parameter during HDFS replication policy creation using the Advanced > File listing threads field.

OPSAPS-65966

Fixed an issue where Ranger policies were not automatically created for Kudu.

OPSAPS-66107

Avoiding unnecessary Resource Manager scheduled refresh in Global Pools Refresh command. During Autoscaling in the public cloud, the scheduled Global Pools refresh command was causing conflicts with the Resource Manager refresh command that is triggered by commission and decommission commands of Yarn service (which caused Autoscale failures as the Resource Manager refresh command was not available). This issue is fixed now.

The repositories for Cloudera Manager 7.6.7-CHF4 are listed in the following table:

https://username:password@archive.cloudera.com/p/cm7/patch/7.6.7-h4-39231315/redhat8/yum

https://username:password@archive.cloudera.com/p/cm7/patch/7.6.7-h4-39231315/redhat8/yum/cloudera-manager.repo

https://username:password@archive.cloudera.com/p/cm7/patch/7.6.7-h4-39231315/redhat7/yum

https://username:password@archive.cloudera.com/p/cm7/patch/7.6.7-h4-39231315/redhat7/yum/cloudera-manager.repo

https://username:password@archive.cloudera.com/p/cm7/patch/7.6.7-h4-39231315/sles12/yum

https://username:password@archive.cloudera.com/p/cm7/patch/7.6.7-h4-39231315/sles12/yum/cloudera-manager.repo

https://username:password@archive.cloudera.com/p/cm7/patch/7.6.7-h4-39231315/ubuntu2004/apt

https://username:password@archive.cloudera.com/p/cm7/patch/7.6.7-h4-39231315/ubuntu2004/apt/cloudera-manager.list

https://username:password@archive.cloudera.com/p/cm7/patch/7.6.7-h4-39231315/ubuntu1804/apt

https://username:password@archive.cloudera.com/p/cm7/patch/7.6.7-h4-39231315/ubuntu1804/apt/cloudera-manager.list