Install and Upgrade Notes

Error reported when upgrading Cloudera Manager Agents

When upgrading to Cloudera Manager 6.3.4, you will see the following error when upgrading the Cloudera Manager Agents:
supervisord version 3.4.0 does not match with SCM version 6.3.4. You must hard restart the agent to switch to the right version of supervisord.

You should ignore or suppress this message. Do not restart the agents.

Cloudera Bug: ENGESC-5109

Installation and Upgrade Changes

Installation or Upgrade of Cloudera Manager and CDH requires authentication to access downloads

Beginning with Cloudera Manager and CDH 6.3.3, downloading new versions of these products will require a valid Cloudera Enterprise license file, and/or a username and password obtained from Cloudera. All Cloudera Manager package, CDH parcel and CDH package repositories now require authentication with valid credentials to access any version numbered 6.3.3 or later. For more information on using these credentials, see the documentation below.

Cloudera Express has been discontinued

Beginning with CDH 6.3.3 (and CDP Data Center 7.0), Cloudera Express is no longer available. Upgrades to Cloudera Manager or CDH 6.3.3 and higher are not supported when running Cloudera Express. A valid Cloudera Enterprise or CDP Data Center license must be in place before upgrading to Cloudera Manager 6.3.3 or 7.x or the upgrade will not be completed.

Downgrading from Cloudera Enterprise license to Cloudera Express license is also no longer supported in Cloudera Manager 6.3.3 and higher.

Upgrades to Cloudera Manager 6.3 Fail with Hive Cloud replication schedules

If you have any Hive Replication Schedules that replicate to a cloud destination, delete these replication schedules before continuing with the upgrade. You can re-create these Replication Schedules after the Cloudera Manager upgrade is complete.

Cloudera Bug: OPSAPS-54117

Upgrades to Cloudera Enterprise 6.x

Using OpenJDK 11 on CDH6.3 and above requires re-installation of YARN MapReduce Framework JARs

Because several Java internal APIs are removed in JDK11, using older versions of MR Framework JARs will fail MR/Hive jobs, with the following error:
```
2019-07-18 14:54:52,483 ERROR [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.lang.NoSuchMethodError: sun.nio.ch.DirectBuffer.cleaner()Lsun/misc/Cleaner;
    at org.apache.hadoop.crypto.CryptoStreamUtils.freeDB(CryptoStreamUtils.java:41)
    at org.apache.hadoop.crypto.CryptoInputStream.freeBuffers(CryptoInputStream.java:687)
    at org.apache.hadoop.crypto.CryptoInputStream.close(CryptoInputStream.java:320)
    at java.base/java.io.FilterInputStream.close(FilterInputStream.java:180)
```
Workaround:
  1. Go to the YARN service.
  2. Select Actions > Install YARN MapReduce Framework JARs and click Install YARN MapReduce Framework JARs
  3. To verify, you will find the new MR Framework JARs under the MR Application Framework Path (default: /user/yarn/mapreduce/mr-framework/) For example:
    ``
    hdfs dfs -ls /user/yarn/mapreduce/mr-framework/
    Found 5 items
    -rw-r--r-- 332 yarn hadoop  215234466 2018-07-19 11:40 /user/yarn/mapreduce/mr-framework/3.0.0-cdh6.0.0-mr-framework.tar.gz
    -rw-r--r--  97 yarn hadoop  263033197 2018-05-18 18:38 /user/yarn/mapreduce/mr-framework/3.0.0-cdh6.0.x-mr-framework.tar.gz
    -rw-r--r-- 331 yarn hadoop  222865312 2018-11-08 14:39 /user/yarn/mapreduce/mr-framework/3.0.0-cdh6.1.0-mr-framework.tar.gz
    -rw-r--r-- 327 yarn hadoop  232020483 2019-02-25 22:46 /user/yarn/mapreduce/mr-framework/3.0.0-cdh6.2.0-mr-framework.tar.gz
    -rw-r--r-- 326 yarn hadoop  234641649 2019-07-23 15:49 /user/yarn/mapreduce/mr-framework/3.0.0-cdh6.3.0-mr-framework.tar.gz
    ```

Cloudera Bug: CDH-81350

Cloudera Manager 6.1 does not substitute {{CONF_DIR}}/library.leveldbjni.path for YARN processes

After upgrading to Cloudera Manager 6.1 or 6.2, if the /tmp directory is mounted using the noexec flag, then starting or restarting YARN NodeManagers will fail for all CDH versions. Note that this change may cause configuration staleness on Cloudera Manager when upgrading to 6.3.0 for clusters running YARN.

Cloudera Bug: OPSAPS-50253

Restart of Impala and Hive required for Cloudera Manager 6.2 upgrade with ADLS

After upgrading to Cloudera Manager 6.2 or higher, Impala and Hive will be marked as stale for users running CDH 6.1 and using the ADLS Service. You will need to restart Hive and Impala before being able to connect to ADLS Gen2, but all previous functionality will continue to work without a restart. The configurations that will be marked stale are:
  • fs.azure.account.auth.type
  • fs.azure.account.oauth.provider.type
  • fs.azure.account.oauth2.client.endpoint
  • fs.azure.account.oauth2.client.id
  • fs.azure.account.oauth2.client.secret.

Cloudera Bug: OPSAPS-47436

TSB-359 Backup and Disaster Recovery (BDR) HDFS and Hive Replications will fail on clusters running Cloudera Manager 6.1.0

Backup and Disaster Recovery (BDR) HDFS and Hive Replications will fail when replicating from secured (Kerberized) source clusters to destination clusters that have been upgraded to Cloudera Manager 6.1.0.

This also affects new installations of Cloudera Manager 6.1.0 on the destination cluster if an admin restarts the Cloudera Manager service.

Products affected: Cloudera Manager Backup and Disaster Recovery in a secure (Kerberized) environment

Releases affected: Cloudera Manager 6.1.0 (when used as the destination cluster of HDFS and/or Hive replication)

Users affected: Customers using HDFS or Hive Replication

Severity (Low/Medium/High): High

Root Cause and Impact:

In HDFS and Hive Replication, Cloudera Manager first runs a process on the destination cluster to verify if the replication is possible. Due to a bug, the source cluster is treated as an insecure (non-kerberized) cluster. As a result, replication fails.

You will see the exception javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)] in the process stderr logs.

Immediate action required: If you use BDR, do not upgrade a destination cluster to Cloudera Manager 6.1.0. Upgrade to Cloudera Manager 6.1.1 or higher when it becomes available.

If you have already upgraded your destination cluster to Cloudera Manager to 6.1.0, use the following workaround:

  1. For an existing HDFS or Hive replication schedule, select Actions > Edit Configuration.
  2. Save the schedule.

Please note that you will need to edit only one schedule even if you have multiple schedules.

Note: This workaround is not persistent. That is, if you restart the Cloudera Manager service, you must repeat the above workaround.

Cloudera Issue: OPSAPS-48865

Fixed in Cloudera Manager 6.1.1

Upgrades from Cloudera Enterprise 5.15 or 5.16 to 6.0x are not supported

You cannot upgrade to Cloudera Manager or CDH 6.0.0 from Cloudera Manager or CDH 5.15 or 5.16.

Upgrading to CDH 6.1.0 Enables Direct SQL mode in Hive service by default

For details about the Cloudera Manager Enable Direct SQL option, refer to Hive Metastore Database.

Upgrades from Cloudera Enterprise 6.0 Beta Release to 6.x General Release Not supported

You cannot upgrade to any Cloudera Manager or CDH 6.x general release from the Cloudera Manager or CDH 6.0 Beta release.

Cloudera Express License Enforcement

Use of Cloudera Express is limited to a total of 100 hosts running CDH6.0 or later across all environments used by an organization..

Note the following:
  • Cloudera Manager will not allow you to add hosts to a CDH 6.x cluster if the total number of hosts across all CDH 6.x clusters will exceed 100.
  • Cloudera Manager will not allow you to upgrade any cluster to CDH 6.x if the total number of managed CDH6.x cluster hosts will exceed 100. If an upgrade from Cloudera Manager 6.0 to 6.1 fails due to this limitation, you must downgrade Cloudera Manager to version 6.0, remove some hosts so that the number of hosts is less than 100, then retry the upgrade.

Affected Versions: CM 6.1 and higher

Cloudera Issue: OPSAPS-46868

Cloudera Data Science Workbench is Not Supported with Cloudera Enterprise 6.0

Cloudera Data Science Workbench is not supported with Cloudera Enterprise 6.0.x. Cloudera Data Science Workbench 1.5.0 (and higher) is supported with Cloudera Manager 6.1.x (and higher) and CDH 6.1.x (and higher).

Impala roles with SELECT or INSERT privileges receive REFRESH privileges during the upgrade

Due to the Sentry and Impala fine grained privileges feature in 5.16.0, if a role has the SELECT or INSERT privilege on an object in Impala before upgrading to CDH 5.16.0, that role will automatically get the REFRESH privilege during the upgrade.

Hue requires manual installation of psycopg2

If you are installing or upgrading to CDH 6.0.0 and using the PostgreSQL database for the Hue database, you must install psycopg2 2.5.4 or higher on all Hue hosts. SeeInstalling the psycopg2 Python package.

Cloudera Issue: OPSAPS-47080

Solr service with no added collections causes the upgrade process to fail

CDH 5.x to CDH 6.x upgrade fails while performing the bootstrap collections step of the solr-upgrade.sh script with the error message:
Failed to execute command Bootstrap Solr Collections on service Solr
if there are no collections present in Solr.

Workaround: If there are no collections added to it, remove the Solr service from your cluster before you start the upgrade.

Affected Versions: CDH 6.0.0, 6.0.1, 6.1.0, 6.1.1, 6.2.0, 6.2.1, 6.3.0, 6.3.1, 6.3.2

Fixed Version: CDH 6.3.3

Cloudera Issue: CDH-82042

CDH Upgrade fails to delete Solr data from HDFS

The CDH upgrade process fails to delete Solr data from HDFS and the recreated collections fail to be initialized due to the existing indexes.

Workaround: Perform the following steps after you run the CDH Upgrade wizard and before you finalize the HDFS upgrade:
  1. Log in to the Cloudera Manager Admin Console.
  2. Go to the Solr service page.
  3. Stop the Solr service and dependent services. Click Actions > Stop.
  4. Click Actions > Reinitialize Solr State for Upgrade.
  5. Click Actions > Bootstrap Solr Configuration.
  6. Start the Solr and dependent services. Click Actions > Start.
  7. Click Actions > Bootstrap Solr Collections.

Affected Versions: CDH 6.0.0

Fixed Versions: Cloudera Manager 6.0.1

Cloudera Issue: OPSAPS-47502

Package Installation of CDH Fails

When you install CDH with packages from a custom repository, ensure that the version of CDH you select for Select the version of CDH matches the version of CDH for the custom repository. Selecting the CDH version and specifying a custom repository are done during the Select Repository stage of installation.

If the versions do not match, installation fails.

Affected Versions: Cloudera Manager 6.x

Fixed Versions: N/A

Apache Issue: N/A

Cloudera Issue: OPSAPS-45703

Uninstall CDH 5 Sqoop connectors for Teradata and Netezza before upgrading to CDH 6

Sqoop includes two connectors, one for Teradata and one for Netezza. The connectors are released in separate parcels and tarballs and can be installed in Cloudera Manager or manually. The versioning of the connectors takes the form <connector_version>c<major_cdh_version>. For example, 1.6c5 refers to the connector 1.6 for CDH 5. The manifest files do not prohibit installing the CDH 5 connectors on CDH 6, but they are not compatible with CDH 6.

If you have CDH 5 connectors installed, they will not be automatically upgraded during the CDH upgrade, and they are not compatible with CDH 6, so they should be uninstalled before the upgrade. Keeping the CDH 5 connectors will not cause the upgrade to fail, but instead will cause a failure to occur during Sqoop runtime. Cloudera will release the connectors for CDH 6 at a later time.

For more information about the Teradata and Netezza connectors, go to Cloudera Enterprise Connector Documentation and choose the connector and version to see the documentation for your connector.

Unsupported Sqoop options cause upgrade failures

New fail-fast checks for unsupported options were introduced in CDH 6. Users should check the jobs stored in their Sqoop metastore and remove all unsupported options. Some unsupported options were silently ignored in earlier CDH versions during upgrades, but in CDH 6, the same options fail instantly. See the following JIRAs in Apache Sqoop Incompatible Changes:

Generated Avro code from CDH 5 should be regenerated when upgrading

Changes in logical types cause code generated in Avro with CDH 6 to differ from code generated in Avro with CDH 5. This means that old generated code will not necessarily work in CDH 6. Cloudera recommends that users regenerate their generated Avro code when upgrading.

Upgrading Apache Parquet to CDH 6

Parquet packages and the project’s group ID were renamed, and some of the class methods were removed.

If you directly consumes the Parquet API instead of using Parquet through a CDH component, your need to update and recompile your code. See Parquet API Change for details of the changes.

No HBase Replication Peer Configuration Change During Rolling Update

When doing a rolling upgrade from a CDH 6 version to a higher version, do not do any replication peer configuration changes. This includes removing a peer, adding a peer, and changing the configuration on a peer.

Oracle Database Initialization

Before upgrading from CDH 5 to CDH 6, check the value of the COMPATIBLE initialization parameter in the Oracle Database using the following SQL query: 
SELECT name, value FROM v$parameter WHERE name = 'compatible'
The default value is 12.2.0. If the parameter has a different value, you can set it to the default as shown in the Oracle Database Upgrade Guide.

TLS Protocol Error with OpenJDK

If you are using an older version of OpenJDK 1.8 and have enabled SSL/TLS for the Cloudera Manager Admin Console, you may encounter a TLS protocol error when connecting to the Admin Console, stating that there are no ciphers in common. This is because older versions of OpenJDK may not implement certain TLS ciphers, causing an inability to log into the Cloudera Manager Admin Console when TLS is enabled.

Workaround:

You can workaround this issue by doing one of the following:
  • Upgrade OpenJDK to a supported version of OpenJDK that is higher than version 1.8.0_181.
  • If it is not possible to upgrade OpenJDK, enable less secure TLS ciphers in Cloudera Manager. You can do this by opening the /etc/default/cloudera-scm-server in a text editor and adding the following line:
    export CMF_OVERRIDE_TLS_CIPHERS=<cipher_list>
    Where <cipher_list> is a list of TLS cipher suites separated by colons. For example:
    export CMF_OVERRIDE_TLS_CIPHERS="TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256:TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256:TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384:TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384:TLS_DHE_RSA_WITH_AES_128_GCM_SHA256:TLS_DHE_RSA_WITH_AES_256_GCM_SHA384:TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256:TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256:TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA:TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384:TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA:TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384:TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA:TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA:TLS_DHE_RSA_WITH_AES_128_CBC_SHA256:TLS_DHE_RSA_WITH_AES_128_CBC_SHA:TLS_DHE_RSA_WITH_AES_256_CBC_SHA256:TLS_DHE_RSA_WITH_AES_256_CBC_SHA:TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA:TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA:TLS_EDH_RSA_WITH_3DES_EDE_CBC_SHA:TLS_RSA_WITH_AES_128_GCM_SHA256:TLS_RSA_WITH_AES_256_GCM_SHA384:TLS_RSA_WITH_AES_128_CBC_SHA256:TLS_RSA_WITH_AES_256_CBC_SHA256:TLS_RSA_WITH_AES_128_CBC_SHA:TLS_RSA_WITH_AES_256_CBC_SHA:TLS_RSA_WITH_3DES_EDE_CBC_SHA"
    

Cloudera Bug: OPSAPS-49578

Restart Kafka after upgrading Cloudera Manager

After upgrading Cloudera Manager, Kafka will be marked as stale. At your next opportunity, please restart the Kafka service to allow these new metrics to be collected and new configurations to be effective. The configurations that will be marked stale are:
  • num.network.threads=8
  • num.recovery.threads.per.data.dir=1
  • num.replica.fetchers=4
  • producer.metrics.enable

Cloudera Bug: OPSAPS-49741

Upgrade Failure: active NameNode not found

On a Cloudera Manager cluster host running on either CentOS 7.0 or 7.1, performing a curl operation to access the NameNode web UI results in an error similar to the following if the web UI is SSL-enabled:

curl https://nn1.example.com:20102
curl: (35) Cannot communicate securely with peer: no common encryption algorithm(s).

This issue occurs on Cloudera Manager clusters running versions 6.1 and higher that exclude insecure ciphers by default. In addition, an SSL bug with curl on CentOS versions 7.0 and 7.1 prevents negotiation of a more secure SSL cipher with the HDFS NameNode web UI.

Workaround: Update yum packages to the latest available versions by running the following command:

yum update nss-util nss-sysinit nss-tools

Alternate workaround for Cloudera Manager 6.0: Specify the cipher by running the following command:
curl --tlsv1 --ciphers rsa_aes_256_cbc_sha_256 -k -v <hostname>:<port>
Alternate Workaround for Cloudera Manager 6.1 or higher:
  1. Log into the Cloudera Manager Admin Console.
  2. Go to the HDFS Service
  3. Select the Configuration tab.
  4. Locate the SSL/TLS Cipher Suite property
  5. Select Intermediate 2018 (Needed for legacy clients or Redhat 6).
  6. Restart the HDFS service

Cloudera Bug: CDH-81328

Upgrades to Cloudera Enterprise 5.x

Flume Kafka client incompatible changes in CDH 5.8

Due to the change of offset storage from ZooKeeper to Kafka in the CDH 5.8 Flume Kafka client, data might not be consumed by the Flume agents, or might be duplicated (if kafka.auto.offset.reset=smallest) during an upgrade to CDH 5.8.

Cloudera Issue: TSB-173

Upgrade to CDH 5.13 or higher Requires Pre-installation of Spark 2.1 or Spark 2.2

If your cluster has Spark 2.0 or Spark 2.1 installed and you want to upgrade to CDH 5.13 or higher, you must first upgrade to Spark 2.1 release 2 or later before upgrading CDH. To install these versions of Spark, do the following before running the CDH Upgrade Wizard:

  1. Install the Custom Service Descriptor (CSD) file. See
  2. Download, distribute, and activate the Parcel for the version of Spark that you are installing:
    • Spark 2.1 release 2: The parcel name includes "cloudera2" in its name.
    • Spark 2.2 release 1: The parcel name includes "cloudera1" in its name.
    See Managing Parcels.

Affected versions: CDH 5.13.0 and higher

Cloudera Issue: CDH-56775

Sentry may require increased Java heap settings before upgrading CDH to 5.13

Before upgrading to CDH 5.13 or higher, you may need to increase the size of the Java heap for Sentry. A warning will be displayed during upgrade, but it its the user's responsibility to ensure this setting is adjusted properly before proceeding. See Performance Guidelines.

Affected versions: CDH 5.13 or higher

Cloudera Issue: OPSAPS-42541

Apache MapReduce Jobs May Fail During Rolling Upgrade to CDH 5.11.0 or CDH 5.11.1

In CDH 5.11, Cloudera introduced four new counters that are reported by MapReduce jobs. During a rolling upgrade from a cluster running CDH 5.10.x or lower to CDH 5.11.0 or CDH5.11.1, a MapReduce job with an application master running on a host running CDH 5.10.x or lower may launch a map or reduce task on one of the newly-upgraded CDH 5.11.0 or CDH 5.11.1 hosts. The new task will attempt to report the new counter values, which the old application master will not understand, causing an error in the logs similar to the following:
2017-06-08 17:43:37,173 WARN [Socket Reader #1 for port 41187]
org.apache.hadoop.ipc.Server: Unable to read call parameters for client 10.17.242.22on
connection protocol org.apache.hadoop.mapred.TaskUmbilicalProtocol for rpcKind
RPC_WRITABLE
java.lang.ArrayIndexOutOfBoundsException: 23
   at
...

This error could cause the task and the job to fail.

Workaround:

Avoid performing a rolling upgrade to CDH 5.11.0 or CDH 5.11.1 from CDH 5.10.x or lower. Instead, skip CDH 5.11.0 and CDH 5.11.1 if you are performing a rolling upgrade, and upgrade to CDH 5.12 or higher, or CDH 5.11.2 or higher when the release becomes available.

Cloudera Issue: DOCS-2384, TSB-241

Cloudera Manager set catalogd default jvm memory to 4G can cause out of memory error on upgrade to Cloudera Manager 5.7 or higher

After upgrading to 5.7 or higher, you might see a reduced Java heap maximum on Impala Catalog Server due to a change in its default value. Upgrading from Cloudera Manager lower than 5.7 to Cloudera Manager 5.8.2 no longer causes any effective change in the Impala Catalog Server Java Heap size.

When upgrading from Cloudera Manager 5.7 or later to Cloudera Manager 5.8.2, if the Impala Catalog Server Java Heap Size is set at the default (4GB), it is automatically changed to either 1/4 of the physical RAM on that host, or 32GB, whichever is lower. This can result in a higher or a lower heap, which could cause additional resource contention or out of memory errors, respectively.

Cloudera Issue: OPSAPS-34039