Rolling Back a Cloudera Private Cloud Base Upgrade from versions 7.1.1 - 7.1.7 to CDH 6

You cannot roll back HDFS while high availability is enabled. The rollback procedure in this topic creates a temporary configuration without high availability. Regardless of whether high availability is enabled, follow the steps in this section.

1. Roll back all of the Journal Nodes. (Only required for clusters where high availability is enabled for HDFS). Use the JournalNode backup you created when you backed up HDFS before upgrading to Cloudera Private Cloud Base.
   1. Log in to each Journal Node host and run the following commands:

      rm -rf /dfs/jn/ns1/current/*

      cp -rp <Journal_node_backup_directory>/ns1/current/* /dfs/jn/ns1/current/

   2. Start the JournalNodes using Cloudera Manager:
      1. Go to the HDFS service.
      2. Select the Instances tab.
      3. Select all JournalNode roles from the list.
      4. Click Actions for Selected > Start.
2. Roll back all of the NameNodes. Use the NameNode backup directory you created before upgrading to Cloudera Private Cloud Base. (/etc/hadoop/conf.rollback.namenode) to perform the following steps on all NameNode hosts:
   1. (Clusters with TLS enabled only) Edit the /etc/hadoop/conf.rollback.namenode/ssl-server.xml file on all NameNode hosts (located in the temporary rollback directory) and update the keystore passwords with the actual cleartext passwords.
      The passwords will have values that look like this:

      <property>
          <name>ssl.server.keystore.password</name>
          <value>********</value>
        </property>
        <property>
          <name>ssl.server.keystore.keypassword</name>
          <value>********</value>
        </property>
      

   2. (TLS only) Edit the /etc/hadoop/conf.rollback.namenode/ssl-server.xml file and remove the hadoop.security.credential.provider.path property.
3. Edit the /etc/hadoop/conf.rollback.namenode/hdfs-site.xml file on all NameNode hosts and make the following changes:
   1. Update the dfs.namenode.inode.attributes.provider.class property. If Sentry was installed prior to the upgrade, change the value of the property from org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer to "org.apache.sentry.hdfs.SentryINodeAttributesProvider. If Sentry was not installed, remove this property.
   2. Change the path in the dfs.hosts property to the value shown in the example below. The file name, dfs_all_hosts.txt, may have been changed by a user. If so, substitute the correct file name.

      # Original version of the dfs.hosts property:
      <property>
      <name>dfs.hosts</name>
      <value>/var/run/cloudera-scm-agent/process/63-hdfs-NAMENODE/dfs_all_hosts.txt</value>
      </property>

      # New version of the dfs.hosts property:
      <property>
      <name>dfs.hosts</name>
      <value>/etc/hadoop/conf.rollback.namenode/dfs_all_hosts.txt</value>
      </property>

   3. Remove the property that has the following value:

      com.cloudera.navigator.audit.hdfs.HdfsAuditLoggerCdh5

4. Edit the /etc/hadoop/conf.rollback.namenode/core-site.xml and change the value of the net.topology.script.file.name property to /etc/hadoop/conf.rollback.namenode. For example:

   # Original property
   <property>
   <name>net.topology.script.file.name</name>
   <value>/var/run/cloudera-scm-agent/process/63-hdfs-NAMENODE/topology.py</value>
   </property>

   # New property
   <property>
   <name>net.topology.script.file.name</name>
   <value>/etc/hadoop/conf.rollback.namenode/topology.py</value>
   </property>

5. Edit the /etc/hadoop/conf.rollback.namenode/topology.py file and change the value of MAP_FILE to /etc/hadoop/conf.rollback.namenode. For example:

   MAP_FILE = '/etc/hadoop/conf.rollback.namenode/topology.map'

6. (TLS-enabled clusters only) Run the following command:

   sudo -u hdfs kinit hdfs/<NameNode Host name> -l 7d -kt /etc/hadoop/conf.rollback.namenode/hdfs.keytab

7. Run the following command:

   sudo -u hdfs hdfs --config /etc/hadoop/conf.rollback.namenode namenode -rollback

8. Restart the NameNodes and JournalNodes using Cloudera Manager:
   1. Go to the HDFS service.
   2. Select the Instances tab, and then select all Failover Controller, NameNode, and JournalNode roles from the list.
   3. Click Actions for Selected > Restart.
9. Rollback the DataNodes.
   Use the DataNode rollback directory you created before upgrading to Cloudera Private Cloud Base (/etc/hadoop/conf.rollback.datanode) to perform the following steps on all DataNode hosts:

   1. (Clusters with TLS enabled only) Edit the /etc/hadoop/conf.rollback.datanode/ssl-server.xml file on all DataNode hosts (Located in the temporary rollback directory.) and update the keystore passwords (ssl.server.keystore.password and ssl.server.keystore.keypassword) with the actual passwords.
      The passwords will have values that look like this:

      <property>
          <name>ssl.server.keystore.password</name>
          <value>********</value>
        </property>
        <property>
          <name>ssl.server.keystore.keypassword</name>
          <value>********</value>
        </property>
      

   2. (Clusters with TLS only) Edit the /etc/hadoop/conf.rollback.datanode/ssl-server.xml file and remove the hadoop.security.credential.provider.path property.
   3. (Clusters with Kerberos enabled only) Modify the Kerberos file (hdfs.keytab) path in /etc/hadoop/conf.rollback.datanode/core-site.xml and /etc/hadoop/conf.rollback.datanode/hdfs-site.xml files to the absolute path. For example, the absolute path of the Kerberos file can be /etc/hadoop/conf.rollback.datanode/hdfs.keytab.
   4. Edit the /etc/hadoop/conf.rollback.datanode/hdfs-site.xml file and remove the dfs.datanode.max.locked.memory property.
   5. Run one of the following commands:
      • Run as root if the DataNodes use reserved ports.

        Search in the logs for completed rollback line. It will not be shown in the command line of the rollback.

      • If the DataNode is running with privileged ports (usually 1004 and 1006):

        cd /etc/hadoop/conf.rollback.datanode
        export HADOOP_SECURE_DN_USER=hdfs
        export JSVC_HOME=/opt/cloudera/parcels/<parcel_filename>/lib/bigtop-utils
        hdfs --config /etc/hadoop/conf.rollback.datanode datanode -rollback
        

      • If the DataNode is not running on privileged ports:

        cd /etc/hadoop/conf.rollback.datanode
        sudo hdfs --config /etc/hadoop/conf.rollback.datanode datanode -rollback
        

        You may see the following error after issuing these commands:

        ERROR datanode.DataNode: Exception in secureMain
        java.io.IOException: The path component: '/var/run/hdfs-sockets' in '/var/run/hdfs-sockets/dn' has permissions 0755 uid 39998 and gid 1006. 
        It is not protected because it is owned by a user who is not root and not the effective user: '0'.

        The error message will also include the following command to run:

        chown root /var/run/hdfs-sockets

        After running this command, rerun the DataNode rollback command:

        sudo hdfs --config /etc/hadoop/conf.rollback.datanode datanode -rollback

        The DataNodes will now restart successfully.

      When the rolling back of the DataNodes is complete, terminate the console session by typing Control-C. Look for output from the command similar to the following that indicates when the DataNode rollback is complete:

      Rollback of /dataroot/ycloud/dfs/dn/current/BP-<Block Group number> is complete

   6. If High Availability for HDFS is enabled, restart the HDFS service. In the Cloudera Manager Admin Console, go to the HDFS service and select Actions > Restart.
   7. If high availability is not enabled for HDFS, use the Cloudera Manager Admin Console to restart all NameNodes and DataNodes.
      1. Go to the HDFS service.
      2. Select the Instances tab
      3. Select all DataNode and NameNode roles from the list.
      4. Click Actions for Selected > Restart.
10. If high availability is not enabled for HDFS, roll back the Secondary NameNode.
    1. (Clusters with TLS enabled only) Edit the /etc/hadoop/conf.rollback.secondarynamenode/ssl-server.xml file on all Secondary NameNode hosts (Located in the temporary rollback directory.) and update the keystore passwords with the actual cleartext passwords.
       The passwords will have values that look like this:

       <property>
           <name>ssl.server.keystore.password</name>
           <value>********</value>
         </property>
         <property>
           <name>ssl.server.keystore.keypassword</name>
           <value>********</value>
         </property>
       

    2. (TLS only) Edit the /etc/hadoop/conf.rollback.secondarynamenode/ssl-server.xml file and remove the hadoop.security.credential.provider.path property.
    3. Log in to the Secondary NameNode host and run the following commands:

       rm -rf /dfs/snn/*
       cd /etc/hadoop/conf.rollback.secondarynamenode/
       sudo -u hdfs hdfs --config /etc/hadoop/conf.rollback.secondarynamenode secondarynamenode -format
       

       When the rolling back of the Secondary NameNode is complete, terminate the console session by typing Control-C. Look for output from the command similar to the following that indicates when the Secondary NameNode rollback is complete:

       2020-12-21 17:09:36,239 INFO namenode.SecondaryNameNode: Web server init done
       

11. Restart the HDFS service. Open the Cloudera Manager Admin Console, go to the HDFS service page, and select Actions > Restart.

    The Restart Command page displays the progress of the restart. Wait for the page to display the Successfully restarted service message before continuing.