Known Issues and Limitations in Cloudera Manager 6.3.0

Cloudera Manager 6.x issue with the service role Resume

If a selected service role on a node is restarted and fails, and the customer clicks the "Resume" button in Cloudera Manager, the service role on all of the nodes will be restarted concurrently.

Products affected: Cloudera Manager

Releases affected:
  • Cloudera Manager 5.5 and later
  • Cloudera Manager 6.0 until 6.3.3
  • Cloudera Manager 7.1.x

Users affected: Users with admin role in Cloudera Manager can impact end users of the service.

Impact:In production clusters this can result in a cluster-wide service outage; Already observed for the YARN service and the HDFS service in a few clusters.

Severity: High

Action required:
  • A workaround exists where instead of performing a restart we recommend performing a stop/start of the services.
  • Issue is fixed in CM-6.3.4, CM-7.2.1 and above.

Knowledge article: For the latest update on this issue see the corresponding Knowledge article: Cloudera Customer Advisory: Cloudera Manager 6.x issue with service role Resume

Cloudera Manager installation fails on MariaDB 10.2.8 and later

When installing Cloudera Manager using MariaDB 10.2.8 or later, the Cloudera Manager web server doesn't come up and the install process ends with a failed status. The cloudera-scm-server.log includes the following SQL error:

2019-08-28 04:37:10,171 FATAL main:org.hsqldb.cmdline.SqlFile: SQL Error at 'UTF-8' line 57:
  "alter table ROLE_CONFIG_GROUPS
  drop column REVISION_ID"
  Key column 'REVISION_ID' doesn't exist in table
2019-08-28 04:37:10,171 FATAL main:org.hsqldb.cmdline.SqlFile: Rolling back SQL transaction.
2019-08-28 04:37:10,172 ERROR main:com.cloudera.enterprise.dbutil.SqlFileRunner: Exception while executing ddl scripts.
  com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Key column 'REVISION_ID' doesn't exist in table

Note that MariaDB 10.2.8 is provided by default in some operating systems, including SLES 12 SP4.

Workaround: Replace the default MariaDB 10.2.x version with MariaDB 10.2.7.

Affected Versions: MariaDB 10.2.8 and later

Cloudera Issue: OPSAPS-52340

BDR - Hive restore failing during import

When the table filter used during hive cloud restore is different from the table filter used to create the hive cloud backup, the import step fails with the table not found error. Currently it impacts only the cloud restore scenario.

Products affected: Cloudera Manager

Releases affected:
  • Cloudera Manager 5.15, 5.16
  • Cloudera Manager 6.1.x
  • Cloudera Manager 6.2.x
  • Cloudera Manager 6.3.x

Users affected: BDR, Hive cloud restore, where restore uses a subset of tables from the exported tables

Impact:
  • Limited, the hive cloud restore all tables works properly.
  • The hive cloud restore from the hive cloud backup created prior to Cloudera Manager 5.15 would work without any problem.
  • No other BDR functionality is affected.
Immediate action required:
  • Workaround: Not available. Importing specific tables would fail. Impoting ALL tables would continue to work properly.
  • Upgrade: Upgrade to a Cloudera Manager version containing the fix.

Addressed in release/refresh/patch: Cloudera Manager 7.0 and higher versions

TLS does not work for Agent heartbeat

If you use Add Hosts Wizard to add a host and that host does not have the Linux host command line utility installed, the agent config.ini file on that host will have the Cloudera Manager Server's IP address in the "server_host=" line instead of the hostname. This will cause a problem if you enable TLS for agent to server communication because the agent will connect to the Cloudera Manager Server using the IP address and not the hostname. You can fix this problem by manually changing the "server_host=" to have the correct hostname.

Cloudera also recommends that you install the host utility for future use.

Cloudera Bug: OPSAPS-49273

Upgrade Failure: active NameNode not found

On a Cloudera Manager cluster host running on either CentOS 7.0 or 7.1, performing a curl operation to access the NameNode web UI results in an error similar to the following if the web UI is SSL-enabled:

curl https://nn1.example.com:20102
curl: (35) Cannot communicate securely with peer: no common encryption algorithm(s).

This issue occurs on Cloudera Manager clusters running versions 6.1 and higher that exclude insecure ciphers by default. In addition, an SSL bug with curl on CentOS versions 7.0 and 7.1 prevents negotiation of a more secure SSL cipher with the HDFS NameNode web UI.

Workaround: Update yum packages to the latest available versions by running the following command:

yum update nss-util nss-sysinit nss-tools

Alternate workaround for Cloudera Manager 6.0: Specify the cipher by running the following command:
curl --tlsv1 --ciphers rsa_aes_256_cbc_sha_256 -k -v <hostname>:<port>
Alternate Workaround for Cloudera Manager 6.1 or higher:
  1. Log into the Cloudera Manager Admin Console.
  2. Go to the HDFS Service
  3. Select the Configuration tab.
  4. Locate the SSL/TLS Cipher Suite property
  5. Select Intermediate 2018 (Needed for legacy clients or Redhat 6).
  6. Restart the HDFS service

Cloudera Bug: CDH-81328

HDFS Replication with Sentry causes additional NameNode heap usage

While performing HDFS replication on clusters where Sentry is in use on either or both of the source and destination clusters, you must set the value of Run on Peer as Username to be the same as Run as Username. This action ensures that Sentry provided ACL data is not copied to the target cluster, which results in additional usage of NameNode heap in the target cluster.

Cloudera Bug: OPSAPS-50649

Restart Kafka after upgrading Cloudera Manager

After upgrading Cloudera Manager, Kafka will be marked as stale. At your next opportunity, please restart the Kafka service to allow these new metrics to be collected and new configurations to be effective. The configurations that will be marked stale are:
  • num.network.threads=8
  • num.recovery.threads.per.data.dir=1
  • num.replica.fetchers=4
  • producer.metrics.enable

Cloudera Bug: OPSAPS-49741

Limited IE 11 browser support

Many Cloudera Manager wizards, including Installation wizards and Add Service/Role wizards, cannot be completed when using Microsoft Internet Explorer version 11.x. To work around the issue, use another supported browser. See Browser Requirements.

Affected Versions: Cloudera Manager 6.3.0

Cloudera Bug: OPSAPS-51481

Network Performance Inspector Bandwidth test must be manually reset after aborting the command

If you launch the Network Performance Inspector, and then abort the command before it completes, you will need to manually kill the iperf3 utility in order to run the inspector again.

Workaround:

Log in to each host that was selected for inspection and do the following:
  1. Run the following command to find the iperf3 process:
    ps -ef | grep iperf3
    The output will look similar to the following:
    clouder+  2493     1  0 11:11 ?        00:00:05 ./iperf3 -s
    root      8091  1882  0 11:50 pts/0    00:00:00 grep --color=auto iperf3
  2. Locate the process ID for the iperf3 process owned by the clouder+ user, in this example the process ID is 2493.
  3. Run the following command to kill the iperf3 process:
    kill -9 2493

Affected Versions: Cloudera Manager 6.3.0

Cloudera Bug: OPSAPS-50994

Fixed Versions: CM 6.3.1

Cloudera Manager API Failure serving multiple API calls

If Cloudera Manager server logs list multiple warnings while serving multiple Cloudera Manager API calls and fails with HTTP 500 errors, Cloudera recommends that you reduce the level of caching at the hibernate level. The warning messages look similar to the following:
2019-07-13 01:45:55,657 WARN C3P0PooledConnectionPoolManager[identityToken->2skykra31sef453dohob7|2b8bd14b]-AdminTaskTim
 er:com.mchange.v2.async.ThreadPoolAsynchronousRunner: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector
 @20fc6e86 – APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
Workaround:
  1. Log in to the Cloudera Manager server host using ssh.
  2. Stop the Cloudera Manager server.
  3. Edit the /etc/cloudera-scm-server/db.properties file and append it with the following line:
    com.cloudera.cmf.orm.hibernate.c3p0.max_statements=0 
  4. Start the Cloudera Manager server
  5. Navigate to the Database Info page of Cloudera Manager at https://CM_Server:Port/cmf/debug/dbinfo in the Cloudera Manager UI.
  6. Confirm that the max_statements hibernate property is 0.

OPSAPS-51576