4.4. Complete the Stack Upgrade

  1. Because the file system version has now changed you must start the NameNode manually. On the NameNode host:

     su -l $HDFS_USER -c "export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec && /usr/lib/hadoop/sbin/hadoop-daemon.sh start namenode -upgrade" 

    Depending on the size of your system, this step may take up to 10 minutes.

  2. This upgrade can take a long time depending on the number of files you have. You can use tail to monitor the log file so that you can track your progress:

     tail -f /var/log/$HDFS_LOG_FOLDER/hadoop-hdfs-namenode-$HOSTNAME.log 

    Look for lines that confirm the upgrade is complete, one line per name directory, such as Upgrade of /hadoop/hdfe/namenode is complete. You can also look for Registered FSNamesystem State MBean which follows the upgrade of all name directories.

  3. Prepare the NameNode to work with Ambari:

    1. Open the Ambari Web GUI. If it has been open throughout the process, do a hard reset on your browser to force a reload.

    2. On the Services view, click HDFS to open the HDFS service.

    3. Click View Host to open the NameNode host details page.

    4. Use the dropdown menu to stop the NameNode.

    5. On the Services view, restart the HDFS service. Make sure it passes the ServiceCheck. It is now under Ambari's control.

  4. After the DataNodes are started, HDFS exits safemode. To monitor the status:

    sudo su -l $HDFS_USER -c "hdfs dfsadmin -safemode get”

    Depending on the size of your system, this may take up to 10 minutes or so. When HDFS exits safemode, this is displayed as a response to the command:

    Safe mode is OFF
  5. Make sure that the HDFS upgrade was successful. Go through steps 2 and 3 in Section 9.1 to create new versions of the logs and reports. Substitute "new" for "old" in the file names as necessary

  6. Compare the old and new versions of the following:

    • dfs-old-fsck-1.log versus dfs-new-fsck-1.log.

      The files should be identical unless the hadoop fsck reporting format has changed in the new version.

    • dfs-old-lsr-1.log versus dfs-new-lsr-1.log.

      The files should be identical unless the the format of hadoop fs -lsr reporting or the data structures have changed in the new version.

    • dfs-old-report-1.log versus fs-new-report-1.log

      Make sure all DataNodes previously belonging to the cluster are up and running.

  7. Use the Ambari Web Services view to start YARN.

  8. Use the Ambari Web Services view to start MapReduce2.

  9. Upgrade HBase:

    1. Make sure that all HBase components - RegionServers and HBase Master - are stopped.

    2. Use the Ambari Web Services view, start the ZooKeeper service. Wait until the ZK service is up and running.

    3. On the HBase Master host, make these configuration changes:

      1. In HBASE_CONFDIR/hbase-site.xml, set the property dfs.client.read.shortcircuit to false.

      2. In the configuration file, find the value of the hbase.tmp.dir property and make sure that the directory exists and is readable and writeable for the HBase service user and group.

      3. Go to the Upgrade Folder and check in the saved global configuration file named global_<$TAG> for the value of the property hbase_pid_dir and hbase_log_dir. Make sure that the directories are readable and writeable for the HBase service user and group.

        chown -R $HBASE_USER:$HADOOP_GROUP $hbase_pid_dir
        chown -R $HBASE_USER:$HADOOP_GROUP $hbase_log_dir

        Do this on every host where a RegionServer is installed as well as on the HBase Master host.

      4. Check for HFiles in V1 format. HBase 0.96.0 discontinues support for HFileV1. Before the actual upgrade, run the following command to check if there are HFiles in V1 format:

        hbase upgrade -check

        HFileV1 was a common format prior to HBase 0.94. You may see output similar to:

        Tables Processed:
        Count of HFileV1: 2
        Count of corrupted files: 1
        Corrupted Files:
        Count of Regions with HFileV1: 2
        Regions to Major Compact:

        When you run the upgrade check, if “Count of HFileV1” returns any files, start the hbase shell to use major compaction for regions that have HFileV1 format. For example in the sample output above, you must compact the fa02dac1f38d03577bd0f7e666f12812 and ecdd3eaee2d2fcf8184ac025555bb2af regions.

      5. Upgrade HBase. You must be the HBase service user.

        sudo su -l $HBASE_USER -c "hbase upgrade -execute"

        Make sure that the output contains the string "Successfully completed Znode upgrade".

      6. Use the Services view to start the HBase service. Make sure that Service Check passes.

  10. Upgrade Oozie:

    1. On the Services view, make sure YARN and MapReduce2 are running.

    2. Make sure that the Oozie service is stopped.

    3. Upgrade Oozie. You must be the Oozie service user. On the Oozie host:

      sudo su -l $OOZIE_USER -c"/usr/lib/oozie/bin/ooziedb.sh upgrade -run"

      Make sure that the output contains the string "Oozie DB has been upgrade to Oozie version 'OOZIE Build Version'".

    4. Prepare the WAR file:


      The Oozie server must be not running for this step. If you get the message "ERROR: Stop Oozie first", it means the script still thinks it's running. Check, and if needed, remove the process id (pid) file indicated in the output.

      /usr/lib/oozie/bin/oozie-setup.sh prepare-war

      Make sure that the output contains the string "New Oozie WAR file with added".

    5. Modify the following configuration properties in oozie-site.xml. On the Ambari Server, use /var/lib/ambari-server/resources/scripts/configs.sh to inspect and update the configuration properties as described here.

      Table II.6.2. Properties to Modify
      Action Property Name Property Value
      Add oozie.service.URIHandlerService.uri.handlers org.apache.oozie.dependency.FSURIHandler,org.apache.oozie.dependency.HCatURIHandler
      Add oozie.service.coord.push.check.requeue.interval 30000
      Add oozie.services.ext org.apache.oozie.service.PartitionDependencyManagerService,org.apache.oozie.service.HCatAccessorService
      Add/Modify oozie.service.SchemaService.wf.ext.schemas

      shell-action-0.1.xsd,email-action-0.1.xsd,hive-action-0.2.xsd,sqoop-action-0.2.xsd,ssh-action-0.1.xsd,distcp-action-0.1.xsd,shell-action-0.2.xsd,oozie-sla-0.1.xsd,oozie-sla-0.2.xsd [a]

      [a] Use this list if you have not modified the default Ambari values. If you have added custom schemas, make sure they exist after the modification. The schemas being added here are shell-action-0.2.xsd, oozie-sla-0.1.xsd, and oozie-sla-0.2.xsd. You can add these to your existing list.

    6. Replace the content of /user/oozie/share in HDFS. On the Oozie server host:

      1. Extract the Oozie sharelib into a tmp folder.

        mkdir -p /tmp/oozie_tmp
        cp /usr/lib/oozie/oozie-sharelib.tar.gz /tmp/oozie_tmp
        cd /tmp/oozie_tmp
        tar xzvf oozie-sharelib.tar.gz
      2. Back up the/user/oozie/share folder in HDFS and then delete it. If you have any custom files in this folder back them up separately and then add them back after the share folder is updated.

        su -l $HDFS_USR -c "$hdfs dfs -copyToLocal /user/oozie/share /tmp/oozie_tmp/oozie_share_backup"
        su -l $HDFS_USR -c "$hdfs dfs -rm -r /user/oozie/share"
      3. Add the latest share libs that you extracted in step 1. After you have added the files, modify ownership and acl.

        su -l $HDFS_USR -c "hdfs dfs -copyFromLocal /tmp/oozie_tmp/share /user/oozie/."
        su -l $HDFS_USR -c "hdfs dfs -chown -R oozie:hadoop /user/oozie" 
        su -l $HDFS_USR -c "hdfs dfs -chmod -R 755 /user/oozie"
    7. Use the Services view to start the Oozie service. Make sure that ServiceCheck passes for Oozie.

  11. Update webhcat.

    1. Modify the webhcat-site config type.

      On the Ambari server, use /var/lib/ambari-server/resources/scripts/configs.sh to modify configuration properties in templeton.storage.class:

      For example: configs.sh set <yourhostname> <yourclustername> webhcat-site "templeton.storage.class" "org.apache.hive.hcatalog.templeton.tool.ZooKeeperStorage"
    2. Update the Pig and Hive tar bundles, by updating the following files:

      • /apps/webhcat/pig.tar.gz

      • /apps/webhcat/hive.tar.gz


        You will find these files on a host where webhcat is installed.

      For example, to update a *.tar.gz file:

      1. Move the file to a local directory.

        su -l $HCAT_USR -c "hadoop --config /etc/hadoop/conf fs -copyToLocal /apps/webhcat/*.tar.gz ${local_backup_dir}"

      2. Remove the old file.

        su -l $HCAT_USR -c "hadoop --config /etc/hadoop/conf fs -rm /apps/webhcat/*.tar.gz"

      3. Copy the new file.

        su -l $HCAT_USR -c "hadoop --config /etc/hadoop/conf fs -copyFromLocal /user/share/HDP-webhcat/*.tar.gz /apps/webhcat"

    3. Update /app/webhcat/hadoop-streaming.jar file.

      1. Move the file to a local directory.

        su -l $HCAT_USR -c "hadoop --config /etc/hadoop/conf fs -copyToLocal /apps/webhcat/hadoop-streaming*.jar ${local_backup_dir}"

      2. Remove the old file.

        su -l $HCAT_USR -c "hadoop --config /etc/hadoop/conf fs -rm /apps/webhcat/hadoop-streaming*.jar"

      3. Copy the new hadoop-streaming.jar file.

        su -l $HCAT_USR -c "hadoop --config /etc/hadoop/conf fs -copyFromLocal /user/lib/hadoop-mapreduce/hadoop-streaming*.jar /apps/webhcat"
  12. Make sure Ganglia no longer attempts to monitor JobTracker.

    1. Make sure Ganglia is stopped.

    2. Log into the host where JobTracker was installed (and where ResourceManager is installed after the upgrade).

    3. Backup the folder /etc/ganglia/hdp/HDPJobTracker .

    4. Remove the folder /etc/ganglia/hdp/HDPJobTracker.

    5. Remove the folder $ganglia_runtime_dir/HDPJobTracker.


      For the value of $ganglia_runtime_dir, in the Upgrade Folder, check the saved global configuration file global_<$TAG>.

  13. Use the Services view to start the remaining services back up.

  14. The upgrade is now fully functional but not yet finalized. Using the finalize comand removes the previous version of the NameNode and DataNode's storage directories.


    After the upgrade is finalized, the system cannot be rolled back. Usually this step is not taken until a thorough testing of the upgrade has been performed.

    The upgrade must be finalized before another upgrade can be performed.


    Directories used by Hadoop 1 services set in /etc/hadoop/conf/taskcontroller.cfg are not automatically deleted after upgrade. Administrators can choose to delete these directories after the upgrade.

    To finalize the upgrade:

    sudo su -l $HDFS_USER -c "hadoop dfsadmin -finalizeUpgrade"

    where $HDFS_USER is the HDFS Service user (by default, hdfs).

loading table of contents...