Troubleshooting Cluster Utilization Report issues

This topic describes common Cluster Utilization Report issues and suggested solutions.The following are possible workarounds for issues that arise while generating cluster utilization reports.

YARN Container Usage Aggregation failing on Compute Cluster

In a Compute Cluster setup, while enabling the YARN Container Usage Aggregation report in Cloudera Manager for YARN can cause the yarn_enable_container_usage_aggregation job to fail on the Service Monitor side.

The permission error occurs on Service Monitor while running the aggregation job. The job lacks the necessary HDFS write permissions on the designated directory. This prevents the job from completing, as the user attempting to write the report data (for example, hue) is denied access to a directory owned by another user (for example, hdfs).

The following error message explicitly points to a user (hue) attempting to WRITE to a directory (/mc/1546336759/fs/user) where it lacks the necessary permissions. The HDFS path is owned by hdfs: supergroup with drwxr-xr-x permissions, which means only the hdfs user has write access.

14:37:03,810 WARN com.cloudera.cmon.firehose.polling.CdhTask: 
Exception in doWork for task: yarn2_YARN_USAGE_AGGREGATION_SUB_TASK_yarn2
org.apache.hadoop.security.AccessControl
Exception: Permission denied: user=hue, access=WRITE, inode="/mc/1546336759/fs/user":hdfs:supergroup:drwxr-xr-x
	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:553)
	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:393)

The hue user is an example of a service user that might run this aggregation job, and it must have write access to the designated HDFS path. This issue is typically resolved by adjusting the permissions on the HDFS directory or configuring the job to use an account that already has the correct permissions.

Workaround:

To resolve the permission issue, do the following step to create the required directory in HDFS and assign ownership to the appropriate user:

hdfs dfs -mkdir /mc/<cluster-id>/fs/user/<user>
hdfs dfs -chown <user>  /mc/<cluster-id>/fs/user/<user>

Replace <cluster-id> with the actual cluster ID.

Replace <user> with the Service Monitor job execution user (for example, hue).