Before You Begin

• Read the Cloudera Manager 5 Release Notes.
• Make sure there are no Oozie workflows in RUNNING or SUSPENDED status; otherwise the Oozie database upgrade will fail and you will have to reinstall CDH 4 to complete or kill those running workflows.
• Run the Host Inspector and fix every issue.
• If using security, run the Security Inspector.
• Run hdfs fsck / and hdfs dfsadmin -report and fix any issues.
• If using HBase:
  • Run hbase hbck to make sure there are no inconsistencies.
  • Before you can upgrade HBase from CDH 4 to CDH 5, your HFiles must be upgraded from HFile v1 format to HFile v2, because CDH 5 no longer supports HFile v1. The upgrade procedure itself is different if you are using Cloudera Manager or the command line, but has the same results. The first step is to check for instances of HFile v1 in the HFiles and mark them to be upgraded to HFile v2, and to check for and report about corrupted files or files with unknown versions, which need to be removed manually. The next step is to rewrite the HFiles during the next major compaction. After the HFiles are upgraded, you can continue the upgrade. To check and upgrade the files:
    1. In the Cloudera Admin Console, go to the HBase service and run Actions > Check HFile Version.
    2. Check the output of the command in the stderr log.
       Your output should be similar to the following:

       Tables Processed:
       hdfs://localhost:41020/myHBase/.META.
       hdfs://localhost:41020/myHBase/usertable
       hdfs://localhost:41020/myHBase/TestTable
       hdfs://localhost:41020/myHBase/t
       
       Count of HFileV1: 2
       HFileV1:
       hdfs://localhost:41020/myHBase/usertable /fa02dac1f38d03577bd0f7e666f12812/family/249450144068442524
       hdfs://localhost:41020/myHBase/usertable /ecdd3eaee2d2fcf8184ac025555bb2af/family/249450144068442512
       
       Count of corrupted files: 1
       Corrupted Files:
       hdfs://localhost:41020/myHBase/usertable/fa02dac1f38d03577bd0f7e666f12812/family/1
       Count of Regions with HFileV1: 2
       Regions to Major Compact:
       hdfs://localhost:41020/myHBase/usertable/fa02dac1f38d03577bd0f7e666f12812
       hdfs://localhost:41020/myHBase/usertable/ecdd3eaee2d2fcf8184ac025555bb2af

       In the example above, you can see that the script has detected two HFile v1 files, one corrupt file and the regions to major compact.
    3. Trigger a major compaction on each of the reported regions. This major compaction rewrites the files from HFile v1 to HFile v2 format. To run the major compaction, start HBase Shell and issue the major_compact command.

       $ bin/hbase shell
       hbase> major_compact 'usertable'

       You can also do this in a single step by using the echo shell built-in command.

       $ echo "major_compact 'usertable'" | bin/hbase shell

• Review the upgrade procedure and reserve a maintenance window with enough time allotted to perform all steps. For production clusters, Cloudera recommends allocating up to a full day maintenance window to perform the upgrade, depending on the number of hosts, the amount of experience you have with Hadoop and Linux, and the particular hardware you are using.
• To avoid generating many alerts during the upgrade process, you can enable maintenance mode on your cluster before you start the upgrade. Be sure to exit maintenance mode when you have finished the upgrade, in order to re-enable Cloudera Manager alerts.

Download, Distribute, and Activate Parcels

1. In the Cloudera Manager Admin Console, click the Parcels indicator in the top navigation bar ( or ) to go to the Parcels page.
2. In the parcels page, click Download for the version(s) you want to download. If the parcel you want is not shown here — for example, you want to upgrade to version of CDH that is not the most current version — you can make additional parcel repos available through the parcel settings page.
   • CDH 4 - If you want to run both CDH, Cloudera Impala, and Cloudera Search you should download the CDH, Impala, and Solr parcels. You can find the locations of the previous CDH 4 parcels at http://archive.cloudera.com/cdh4/parcels/. If you are upgrading to CDH 4.3 and want to use Sentry, you can find the separate Sentry parcel at http://archive.cloudera.com/sentry/parcels/latest/). Sentry is included with CDH 4.4 or later parcels.
   • CDH 5 - Download the CDH parcel.
   If your Cloudera Manager server does not have Internet access, you can obtain the required parcel file(s) and put them into the local repository. See Creating and Using a Parcel Repository for more details.
3. When the download has completed, click Distribute for the version you downloaded.
4. When the parcel has been distributed and unpacked, the button will change to say Activate. If you are doing a major version upgrade (that is, from CDH 4 to CDH 5 using parcels), after the distribution phase the button will be labeled Upgrade rather than Activate. In this case, follow the instructions at Upgrading from CDH 4 to CDH 5 Parcels.
5. Click Activate. You are asked if you want to restart the cluster. Do not restart the cluster at this time.
6. Click Close.

Upgrade the Hive Metastore Database

Required if you are upgrading from an earlier version of CDH 4 to CDH 4.2 or later.

1. Make a backup copy of your Hive metastore database.
2. Go to the Hive service.
3. Select Actions > Stop and click Stop to confirm.
4. Select Actions > Upgrade Hive Metastore Database Schema and click Upgrade Hive Metastore Database Schema to confirm.
5. If you have multiple instances of Hive, perform the upgrade on each metastore database.

Upgrade the Oozie ShareLib

1. Go to the Oozie service.
2. Select Actions > Stop and click Stop to confirm.
3. Select Actions > Install Oozie ShareLib and click Install Oozie ShareLib to confirm.
4. When the command completes, click Close.

Upgrade Sqoop

1. Go to the Sqoop service.
2. Select Actions > Stop and click Stop to confirm.
3. Select Actions > Upgrade Sqoop and click Upgrade Sqoop to confirm.
4. When the command completes, click Close.

Restart the Services

1. On the Home page, click to the right of the cluster name and select Restart.
2. Click the Restart button in the confirmation pop-up that appears. The Command Details window shows the progress of starting services.

Deploy the New Client Configuration Files

1. On the Home page, click to the right of the cluster name and select Deploy Client Configuration.
2. Click the Deploy Client Configuration button in the confirmation pop-up that appears.

Remove the Previous CDH 4 Version Packages

If your previous installation of CDH 4 (4.0.x or 4.1.x) was done using packages, you must remove those packages on all hosts on which you installed the parcels and refresh the symlinks so that clients will run the new software versions. Skip this step if your previous installation was using parcels.

1. If Hue is configured to use SQLite as its database:
   1. Stop the Hue service.
   2. Back up the desktop.db to a temporary location before deleting the old Hue Common package. The location of the database can be found in the Hue service Configuration tab under Service > Database > Hue's Database Directory.
     Important: Removing the Hue Common package will remove your Hue database; if you do not back it up you may lose all your Hue user account information.
2. Uninstall the CDH packages on each host:
   • Not including Impala and Search

     Operating System	Command
     RHEL	$ sudo yum remove hadoop hue-common bigtop-jsvc bigtop-tomcat
     SLES	$ sudo zypper remove hadoop hue-common bigtop-jsvc bigtop-tomcat
     Ubuntu or Debian	$ sudo apt-get purge hadoop hue-common bigtop-jsvc bigtop-tomcat

   • Including Impala and Search

     Operating System	Command
     RHEL	$ sudo yum remove hadoop hue-common impala-shell solr-server 'bigtop-*'
     SLES	$ sudo zypper remove hadoop hue-common impala-shell solr-server 'bigtop-*'
     Ubuntu or Debian	$ sudo apt-get purge hadoop hue-common impala-shell solr-server 'bigtop-*'

Update Symlinks for the Newly Installed Components

$ sudo service cloudera-scm-agent restart

Restore Backed up Hue Database

1. In the Cloudera Manager Admin Console, select the Hue service.
2. Select Actions > Stop and click Stop to confirm.
3. Copy the backup to the newly created Hue database directory: /opt/cloudera/parcels/CDH-4.x.0-x.cdh4.x.0.p0.xx/share/hue/desktop.
4. Restart the Hue service.