Upgrading the CDH Cluster

Before you upgrade a CDH cluster, back up Cloudera Manager. Even if you just backed up Cloudera Manager before an upgrade, you should now back up your upgraded Cloudera Manager deployment. See Backing Up Cloudera Manager.

To avoid unnecessary alerts during the upgrade process, enter maintenance mode on your cluster before you start the upgrade. Entering maintenance mode stops email alerts and SNMP traps from being sent, but does not stop checks and configuration validations. Be sure to exit maintenance mode when you have finished the upgrade to re-enable Cloudera Manager alerts. More Information.

On the Home > Status tab, click next to the cluster name and select Enter Maintenance Mode.

1. If you are using packages, or did not choose Upgrade from the parcels page, You can get to the Upgrade CDH page from the Home > Status tab, click next to the cluster name and select Upgrade Cluster.
   Select the previously download/distributed CDH version. If no qualifying CDH parcels are pre-listed, or you want to upgrade to a different version of CDH:

   1. Click the Remote Parcel Repository URLs link and add the appropriate parcel URL. See Parcel Configuration Settings for more information.
   2. Click the Cloudera Manager logo to return to the Home page.
   3. From the Home > Status tab, click next to the cluster name and select Upgrade Cluster.

   If you were previously using packages and would like to switch to using parcels, select Use Parcels.

2. Cloudera Manager 5.14 and lower:

   1. In the Choose CDH Version (Parcels) section, select the CDH version that you want to upgrade to.
   2. Click Continue.

      A page displays the version you are upgrading to and asks you to confirm that you have completed some additional steps.

   3. Click Yes, I have performed these steps.
   4. Click Continue.
   5. Cloudera Manager verifies that the agents are responsive and that the correct software is installed. When you see the No Errors Found message, click Continue.

      The selected parcels are downloaded, distributed, and unpacked.

   6. Click Continue.

      The Host Inspector runs. Examine the output and correct any reported errors.

   Cloudera Manager 5.15 and higher:

   1. In the Upgrade to CDH Version drop-down list, select the version of CDH you want to upgrade to.

      The Upgrade Wizard performs some checks on configurations, health, and compatibility and reports the results. Fix any reported issues before continuing.

   2. Click Run Host Inspector.

      The Host Inspector runs. Click Show Inspector Results to view the Host Inspector report (opens in a new browser tab). Fix any reported issues before continuing.

   3. Click Run Service Inspector. Click Show Inspector Results to view the output of the Service Inspector command (opens in a new browser tab). Fix any reported issues before continuing.
   4. Read the notices for steps you must complete before upgrading, select Yes, I have performed theses steps. ... after completing the steps, and click Continue.

      The selected parcels are downloaded, distributed, and unpacked. The Continue button turns blue when this process finishes.

3. If you have a parcel that works with the existing CDH version, the Upgrade Wizard may display a message that this parcel conflicts with the new CDH version.
   1. Configure and download the newer version of this parcel before proceeding.
      1. Open the Cloudera Manager Admin Console from another browser tab, go to the parcels page, and configure the remote parcel repository for the newer version of this parcel.
      2. Download and distribute the newer version of this parcel.
   2. Click the Run All Checks Again button.
   3. Select the option to resolve the conflicts automatically.
   4. Cloudera Manager deactivates the old version of the parcel, activates the new version and verifies that all hosts have the correct software installed.
4. Click Continue.

   The Choose Upgrade Procedure screen displays. Select the upgrade procedure from the following options:

   • Rolling Restart

     Cloudera Manager upgrades services and performs a rolling restart. The Rolling Restart dialog box displays the impact of the restart on various services. Services that do not support rolling restart undergo a normal restart, and are not available during the restart process.

     Configure the following parameters for the rolling restart (optional):

     Roles to include

     Select which roles to restart as part of the rolling restart.

     Batch Size

     Number of roles to include in a batch. Cloudera Manager restarts the worker roles rack-by-rack, in alphabetical order, and within each rack, hosts are restarted in alphabetical order. If you use the default replication factor of 3, Hadoop tries to keep the replicas on at least 2 different racks. So if you have multiple racks, you can use a higher batch size than the default 1. However, using a batch size that is too high means that fewer worker roles are active at any time during the upgrade, which can cause temporary performance degradation. If you are using a single rack, restart one worker node at a time to ensure data availability during upgrade.

     Advanced Options > Sleep between batches

     Amount of time Cloudera Manager waits before starting the next batch. Applies only to services with worker roles.

     Advanced Options > Failed threshold

     The number of batch failures that cause the entire rolling restart to fail. For example, if you have a very large cluster, you can use this option to allow some failures when you are sure that the cluster will still be functional while some worker roles are down.

     Click the Rolling Restart button when you are ready to restart the cluster.

   • Full Cluster Restart

     Cloudera Manager performs all service upgrades and restarts the cluster.

   • Manual Upgrade

     Cloudera Manager configures the cluster to the specified CDH version but performs no upgrades or service restarts. Manually upgrading is difficult and for advanced users only. Manual upgrades allow you to selectively stop and restart services to prevent or mitigate downtime for services or clusters where rolling restarts are not available.

     To perform a manual upgrade: See Upgrading CDH Manually after an Upgrade Failure for the required steps.

5. Click Continue.

   The Upgrade Cluster Command screen displays the result of the commands run by the wizard as it shuts down all services, activates the new parcels, upgrades services, deploys client configuration files, and restarts services and performs a rolling restart of the services that support it.

   If any of the steps fail, correct any reported errors and click the Resume button. Cloudera Manager will skip restarting roles that have already successfully restarted. Alternately, return to the Home > Status tab and then perform the steps in Upgrading CDH Manually after an Upgrade Failure.

6. Click Continue.
   If your cluster was previously installed or upgraded using packages, the wizard may indicate that some services cannot start because their parcels are not available. To download the required parcels:

   1. In another browser tab, open the Cloudera Manager Admin Console.
   2. Select Hosts > Parcels.
   3. Locate the row containing the missing parcel and click the button to Download, Distribute, and then Activate the parcel.
   4. Return to the upgrade wizard and click the Resume button.

      The Upgrade Wizard continues upgrading the cluster.

7. Click Finish to return to the Home page.

If your cluster uses Sentry and an Oracle database, you must manually add the index on the AUTHZ_PATH.AUTHZ_OBJ_ID column if it does not already exist. Adding the index manually decreases the time Sentry takes to get a full snapshot for HDFS sync. Use the following command to add the index:

CREATE INDEX "AUTHZ_PATH_FK_IDX" ON "AUTHZ_PATH" ("AUTHZ_OBJ_ID");

Several components require additional migrations steps after you complete the CDH upgrade:

• Impala – See Impala Upgrade Considerations
• Cloudera Search

  After upgrading to CDH 6, you must re-index your collections, see Re-Indexing Solr Collections After Upgrading to CDH 6.

• Spark – See Apache Spark Post Upgrade Migration Steps.
• MapReduce 1 to MapReduce 2 – See Migrating from MapReduce 1 (MRv1) to MapReduce 2 (MRv2)
• Kudu – See Upgrade Notes for Kudu 1.10 / CDH 6.3
• Kafka
  1. Remove the following properties from the Kafka Broker Advanced Configuration Snippet (Safety Valve) configuration property.
     • Inter.broker.protocol.version
     • log.message.format.version
  2. Save your changes.
  3. Restart the cluster:
     1. On the Home > Status tab, click to the right of the cluster name and select Restart.
     2. Click Restart that appears in the next screen to confirm. If you have enabled high availability for HDFS, you can choose Rolling Restart instead to minimize cluster downtime. The Command Details window shows the progress of stopping services.

        When All services successfully started appears, the task is complete and you can close the Command Details window.

• HBase - When upgrading to CDH 5.16.1, the hbase/thrift configuration gets broken and needs to be fixed.
  1. Ensure that you have an HBase Thrift Server instance.
     If you do not have an HBase Thrift Server, do the following:

     1. Select the HBase service and click the Instances tab.
     2. Click the Add Role Instances tab.
     3. Follow the wizard to add an HBase Thrift Server Role Instance.
  2. Select the Hue service and click the Configuration tab.
  3. Search for hbase.
  4. Ensure that HBase Service and HBase Thrift Server are set to other than none.
  5. If you use Kerberos, do the following:
     1. Select the HBase service and click the Configuration tab.
     2. Search for hbase thrift authentication.
     3. Set HBase Thrift Authentication to one of the following options:
        • auth-conf: authentication, integrity and confidentiality checking
        • auth-int: authentication and integrity checking
        • auth: authentication only
     4. If you use Impersonation, do the following:
        1. Search for hbase thrift.
        2. Ensure that both Enable HBase Thrift Http Server and Enable HBase Thrift Proxy User are checked.
     5. Verify that HBase allows proxy users:
        1. Navigate to the directory /var/run/cloudera-scm-agent/process/<id>-hbase-HBASETHRIFTSERVER.
        2. Check the core-site.xml and verify that HBase is authorized to impersonate someone:

           <property>
           <name>hadoop.proxyuser.hbase.hosts</name>
           <value>*</value>
           </property>
             
           <property>
           <name>hadoop.proxyuser.hbase.groups</name>
           <value>*</value>
           </property>
           

  6. Select the Hue service and click the Configuration tab.
  7. Search for hue_safety_valve.ini.
  8. Find Hue Service Advance Configuration Snippet (Safety Valve) for hue_safety_valve.ini and add the following snippet:

     [hbase]
     hbase_conf_dir={{HBASE_CONF_DIR}}

  9. If you are using CDH 5.15.0 or higher, add the following snippet in the above hbase section:

     thrift_transport=buffered

  10. Restart the HBase and Hue service by clicking the Stale Service Restart icon that is next to the service. It invokes the cluster restart wizard.

If you entered maintenance mode during this upgrade, exit maintenance mode.

On the Home > Status tab, click next to the cluster name and select Exit Maintenance Mode.