When you run the HBase upgrade script on your running CDH 4 cluster in -check mode, its purpose is to look for all occurrences of HFile v1 and rewrite them to HFile v2. This is because HFile v1 is not supported by HBase 0.96 and all existing files in this format must be rewritten before the upgrade.

This is done by major compaction of the region with these files. Major compaction merges and optimizes files for efficient reads. The process converts the older HFile v1 format to an improved and optimized HFile v2. It also detects corrupt files, those that have an undefined major version (neither 1 nor 2), and are most likely not readable at all. Corrupt files should be removed.

The -check mode prints statistics at the end of the run. It also prints the absolute paths of the tables it has scanned, HFile v1 files if any, regions containing such files (to major compact), and corrupted files.

$ bin/hbase upgrade -check

Tables Processed:
hdfs://localhost:41020/myHBase/.META.
hdfs://localhost:41020/myHBase/usertable
hdfs://localhost:41020/myHBase/TestTable
hdfs://localhost:41020/myHBase/t

Count of HFileV1: 2
HFileV1:
hdfs://localhost:41020/myHBase/usertable /fa02dac1f38d03577bd0f7e666f12812/family/249450144068442524
hdfs://localhost:41020/myHBase/usertable /ecdd3eaee2d2fcf8184ac025555bb2af/family/249450144068442512

Count of corrupted files: 1
Corrupted Files:
hdfs://localhost:41020/myHBase/usertable/fa02dac1f38d03577bd0f7e666f12812/family/1
Count of Regions with HFileV1: 2
Regions to Major Compact:
hdfs://localhost:41020/myHBase/usertable/fa02dac1f38d03577bd0f7e666f12812
hdfs://localhost:41020/myHBase/usertable/ecdd3eaee2d2fcf8184ac025555bb2af

In the example above, you can see that the script has detected two HFile v1 files, one corrupt file and the regions to major compact. By default, the script scans the root directory, as defined by hbase.rootdir. In case you want to scan a specific directory, use the --dir option. For example, the following command would scan the /myHBase/testTable directory.

bin/hbase upgrade --check --dir /myHBase/testTable

You should then major compact all the reported regions so as to transform the files from HFile v1 to HFile v2 format. Once all the HFileV1 files are rewritten, that is, the -check command returns a "No HFile v1 found" message, it is then safe to proceed with the upgrade.

Shutdown your CDH 4 HBase cluster before you run the upgrade script in -execute mode.

To shutdown HBase gracefully:

1. Stop the REST and Thrift server and clients, then stop the cluster.
   1. Stop the Thrift server and clients:

      sudo service hbase-thrift stop

      Stop the REST server:

      sudo service hbase-rest stop

   2. Stop the cluster by shutting down the master and the region servers:
      1. Use the following command on the master node:

         sudo service hbase-master stop

      2. Use the following command on each node hosting a region server:

         sudo service hbase-regionserver stop

2. Stop the ZooKeeper Server:

   $ sudo service zookeeper-server stop

  Important: Before you proceed with Step 4, upgrade your CDH 4 cluster to CDH 5. See Upgrading to CDH 5 for instructions.

This step executes the actual upgrade process. It has a verification step which checks whether or not the Master, RegionServer and backup Master znodes have expired. If not, the upgrade is aborted. This ensures no upgrade occurs while an HBase process is still running. If your upgrade is aborted even after shutting down the HBase cluster, retry after some time to let the znodes expire. Default znode expiry time is 300 seconds.

As mentioned earlier, ZooKeeper and HDFS should be available. If ZooKeeper is managed by HBase, then use the following command to start ZooKeeper.

./hbase/bin/hbase-daemon.sh start zookeeper

The upgrade involves three steps:

• Upgrade Namespace: This step upgrades the directory layout of HBase files.
• Upgrade Znodes: This step upgrades /hbase/replication (znodes corresponding to peers, log queues and so on) and table znodes (keep table enable/disable information). It deletes other znodes.
• Log Splitting: In case the shutdown was not clean, there might be some Write Ahead Logs (WALs) to split. This step does the log splitting of such WAL files. It is executed in a “non distributed mode”, which could make the upgrade process longer in case there are too many logs to split. To expedite the upgrade, ensure you have completed a clean shutdown.

$ bin/hbase upgrade -execute

Your output should be similar to the following:

Starting Namespace upgrade
Created version file at hdfs://localhost:41020/myHBase with version=7
Migrating table testTable to hdfs://localhost:41020/myHBase/.data/default/testTable
…..
Created version file at hdfs://localhost:41020/myHBase with version=8
Successfully completed NameSpace upgrade.
Starting Znode upgrade
….
Successfully completed Znode upgrade
Starting Log splitting
…
Successfully completed Log splitting

The output of the -execute command can either return a success message as in the example above, or, in case of a clean shutdown where no log splitting is required, the command would return a "No log directories to split, returning" message. Either of those messages indicates your upgrade was successful.

  Important: Configuration files

• If you install a newer version of a package that is already on the system, configuration files that you have modified will remain intact.
• If you uninstall a package, the package manager renames any configuration files you have modified from <file> to <file>.rpmsave. If you then re-install the package (probably to install a new version) the package manager creates a new <file> with applicable defaults. You are responsible for applying any changes captured in the original configuration file to the new configuration file. In the case of Ubuntu and Debian upgrades, you will be prompted if you have made changes to a file for which there is a new version; for details, see Automatic handling of configuration files by dpkg.

CDH 5 introduces namespaces for HBase tables. As a result of the upgrade, all tables are automatically assigned to namespaces. The root, meta, and acl tables are added to the hbase system namespace. All other tables are assigned to the default namespace.

To move a table to a different namespace, take a snapshot of the table and clone it to the new namespace. After the upgrade, do the snapshot and clone operations before turning the modified application back on.

  Warning: Do not move datafiles manually, as this can cause data corruption that requires manual intervention to fix.