Rolling Back a CDH 5 to CDH 6 Upgrade

1. On the Home > Status tab, click to the right of the cluster name and select Stop.
2. Click Stop in the confirmation screen. The Command Details window shows the progress of stopping services.

   When All services successfully stopped appears, the task is complete and you can close the Command Details window.

Follow these steps only if your cluster was upgraded using Cloudera parcels.

1. Log in to the Cloudera Manager Admin Console.
2. Select Hosts > Parcels.

   A list of parcels displays.

3. Locate the CDH 5 parcel and click Activate. (This automatically deactivates the CDH 6 parcel.) See Activating a Parcel for more information. If the parcel is not available, use the Download button to download the parcel.
4. If you include any additional components in your cluster, such as Search or Impala, click Activate for those parcels.

1. Stop the Cloudera Management Service.
   1. Log in to the Cloudera Manager Admin Console.
   2. Select Clusters > Cloudera Management Service.
   3. Select Actions > Stop.
2. Stop the Cloudera Manager Server.

   RHEL 7, SLES 12, Debian 8, Ubuntu 16.04 and higher

   sudo systemctl stop cloudera-scm-server

   RHEL 5 or 6, SLES 11, Debian 6 or 7, Ubuntu 12.04 or 14.04

   sudo service cloudera-scm-server stop

3. Hard stop the Cloudera Manager agents. Run the following command on all hosts:

   RHEL 7, SLES 12, Debian 8, Ubuntu 16.04 and higher

   sudo systemctl stop supervisord

   RHEL 5 or 6, SLES 11, Debian 6 or 7, Ubuntu 12.04 or 14.04

   sudo service cloudera-scm-agent hard_stop

Follow these steps only if your cluster was upgraded using packages.

Use the backup of CDH that was taken before the upgrade to restore Cloudera Manager Server files and directories. Substitute the path to your backup directory for cm6_cdh5 in the following steps:

1. On the host where the Event Server role is configured to run, restore the Events Server directory from the CM 6/CDH 5 backup.

   rm -rf /var/lib/cloudera-scm-eventserver/*
   cp -rp /var/lib/cloudera-scm-eventserver_cm6_cdh5/* /var/lib/cloudera-scm-eventserver/

2. Remove the Agent runtime state. Run the following command on all hosts:

   rm -rf /var/run/cloudera-scm-agent /var/lib/cloudera-scm-agent/response.avro

   This command may return a message similar to: rm: cannot remove ‘/var/run/cloudera-scm-agent/process’: Device or resource busy. You can ignore this message.

3. On the host where the Service Monitor is running, restore the Service Monitor directory:

   rm -rf /var/lib/cloudera-service-monitor/*
   cp -rp /var/lib/cloudera-service-monitor_cm6_cdh5/* /var/lib/cloudera-service-monitor/

4. On the host where the Host Monitor is running, restore the Host Monitor directory:

   rm -rf /var/lib/cloudera-host-monitor/*
   cp -rp /var/lib/cloudera-host-monitor_cm6_cdh5/* /var/lib/cloudera-host-monitor/

5. Restore the Cloudera Navigator storage directory from the CM 6/CDH 5 backup.

   rm -rf /var/lib/cloudera-scm-navigator/*
   cp -rp /var/lib/cloudera-scm-navigator_cm6_cdh5/* /var/lib/cloudera-scm-navigator/

1. Log in to the Cloudera Manager Server host.

   ssh my_cloudera_manager_server_host

2. Start the Cloudera Manager Server.

   RHEL 7, SLES 12, Debian 8, Ubuntu 16.04 and higher

   sudo systemctl start cloudera-scm-server

   If the Cloudera Manager Server starts without errors, no response displays.

   RHEL 5 or 6, SLES 11, Debian 6 or 7, Ubuntu 12.04 or 14.04

   sudo service cloudera-scm-server start

   You should see the following:

   Starting cloudera-scm-server: [ OK ]

3. Start the Cloudera Manager Agent.

   Run the following commands on all cluster hosts:

   RHEL 7, SLES 12, Debian 8, Ubuntu 16.04 and higher

   sudo systemctl start cloudera-scm-agent

   If the agent starts without errors, no response displays.

   RHEL 5 or 6, SLES 11, Debian 6 or 7, Ubuntu 12.04 or 14.04

   sudo service cloudera-scm-agent start

   You should see the following:

   Starting cloudera-scm-agent: [ OK ]

4. Start the Cloudera Management Service.
   1. Log in to the Cloudera Manager Admin Console.
   2. Select Clusters > Cloudera Management Service.
   3. Select Actions > Start.

1. Using the backup of Zookeeper that you created when backing up your CDH 5.x cluster, restore the contents of the dataDir on each ZooKeeper server. These files are located in a directory specified with the dataDir property in the ZooKeeper configuration. The default location is /var/lib/zookeeper. For example:

   rm -rf /var/lib/zookeeper/*
   cp -rp /var/lib/zookeeper_cm6_cdh5/* /var/lib/zookeeper/

2. Make sure that the permissions of all the directories and files are as they were before the upgrade.
3. Start ZooKeeper using Cloudera Manager.

You cannot roll back HDFS while high availability is enabled. The rollback procedure in this topic creates a temporary configuration without high availability. Regardless of whether high availability is enabled, follow the steps in this section.

1. Roll back all of the Journal Nodes. (Only required for clusters where high availability is enabled for HDFS). Use the JournalNode backup you created when you backed up HDFS before upgrading to CDH 6.
   1. Log in to each Journal Node host and run the following commands:

      rm -rf /dfs/jn/ns1/current/*

      cp -rp /dfs/jn/ns1/previous/* /dfs/jn/ns1/current/

   2. Start the JournalNodes using Cloudera Manager:
      1. Go to the HDFS service.
      2. Select the Instances tab.
      3. Select all JournalNode roles from the list.
      4. Click Actions for Selected > Start.
2. Roll back all of the NameNodes. Use the NameNode backup directory you created before upgrading CDH (/etc/hadoop/conf.rollback.namenode) to perform the following steps on all NameNode hosts:
   1. (Clusters with TLS enabled only) Edit the /etc/hadoop/conf.rollback.namenode/ssl-server.xml file on all NameNode hosts (located in the temporary rollback directory) and update the keystore passwords with the actual cleartext passwords.
      The passwords will have values that look like this:

      <property>
          <name>ssl.server.keystore.password</name>
          <value>********</value>
        </property>
        <property>
          <name>ssl.server.keystore.keypassword</name>
          <value>********</value>
        </property>
      

   2. (TLS only) Edit the /etc/hadoop/conf.rollback.namenode/ssl-server.xml file and remove the hadoop.security.credential.provider.path property.
3. Edit the /etc/hadoop/conf.rollback.namenode/hdfs-site.xml file on all NameNode hosts and make the following changes:
   1. Delete the cloudera.navigator.client.config property.
   2. Delete the dfs.namenode.audit.loggers property.
   3. Change the path in the dfs.hosts property to the value shown in the example below. The file name, dfs_all_hosts.txt, may have been changed by a user. If so, substitute the correct file name.

      # Original version of the dfs.hosts property:
      <property>
      <name>dfs.hosts</name>
      <value>/var/run/cloudera-scm-agent/process/63-hdfs-NAMENODE/dfs_all_hosts.txt</value>
      </property>

      # New version of the dfs.hosts property:
      <property>
      <name>dfs.hosts</name>
      <value>/etc/hadoop/conf.rollback.namenode/dfs_all_hosts.txt</value>
      </property>

   4. Edit the /etc/hadoop/conf.rollback.namenode/core-site.xml and change the value of the net.topology.script.file.name property to /etc/hadoop/conf.rollback.namenode. For example:

      # Original property
      <property>
      <name>net.topology.script.file.name</name>
      <value>/var/run/cloudera-scm-agent/process/63-hdfs-NAMENODE/topology.py</value>
      </property>

      # New property
      <property>
      <name>net.topology.script.file.name</name>
      <value>/etc/hadoop/conf.rollback.namenode/topology.py</value>
      </property>

   5. Edit the /etc/hadoop/conf.rollback.namenode/topology.py file and change the value of MAP_FILE to /etc/hadoop/conf.rollback.namenode. For example:

      MAP_FILE = '/etc/hadoop/conf.rollback.namenode/topology.map'

   6. (TLS-enabled clusters only) Run the following command:

      sudo -u hdfs kinit hdfs/<NameNode Host name> -l 7d -kt /etc/hadoop/conf.rollback.namenode/hdfs.keytab

   7. Run the following command:

      sudo -u hdfs hdfs --config /etc/hadoop/conf.rollback.namenode namenode -rollback

   8. Restart the NameNodes and JournalNodes using Cloudera Manager:
      1. Go to the HDFS service.
      2. Select the Instances tab, and then select all Failover Controller, NameNode, and JournalNode roles from the list.
      3. Click Actions for Selected > Restart.
4. Rollback the DataNodes.
   Use the DataNode rollback directory you created before upgrading CDH (/etc/hadoop/conf.rollback.datanode) to perform the following steps on all DataNode hosts:

   1. (Clusters with TLS enabled only) Edit the /etc/hadoop/conf.rollback.datanode/ssl-server.xml file on all DataNode hosts (Located in the temporary rollback directory.) and update the keystore passwords (ssl.server.keystore.password and ssl.server.keystore.keypassword) with the actual passwords.
      The passwords will have values that look like this:

      <property>
          <name>ssl.server.keystore.password</name>
          <value>********</value>
        </property>
        <property>
          <name>ssl.server.keystore.keypassword</name>
          <value>********</value>
        </property>
      

   2. (TLS only) Edit the /etc/hadoop/conf.rollback.datanode/ssl-server.xml file and remove the hadoop.security.credential.provider.path property.
   3. Edit the /etc/hadoop/conf.rollback.datanode/hdfs-site.xml file and remove the dfs.datanode.max.locked.memory property.
   4. Run the following command:

      cd /etc/hadoop/conf.rollback.datanode
      sudo -u hdfs hdfs --config /etc/hadoop/conf.rollback.datanode datanode -rollback

      After rolling back the DataNodes, terminate the console session by typing Control-C.

   5. If High Availability for HDFS is enabled, restart the HDFS service. In the Cloudera Manager Admin Console, go to the HDFS service and select Actions > Restart.
   6. If high availability is not enabled for HDFS, use the Cloudera Manager Admin Console to restart all NameNodes and DataNodes.
      1. Go to the HDFS service.
      2. Select the Instances tab
      3. Select all DataNode and NameNode roles from the list.
      4. Click Actions for Selected > Restart.
5. If high availability is not enabled for HDFS, roll back the Secondary NameNode.
   1. (Clusters with TLS enabled only) Edit the /etc/hadoop/conf.rollback.secondarynamenode/ssl-server.xml file on all Secondary NameNode hosts (Located in the temporary rollback directory.) and update the keystore passwords with the actual cleartext passwords.
      The passwords will have values that look like this:

      <property>
          <name>ssl.server.keystore.password</name>
          <value>********</value>
        </property>
        <property>
          <name>ssl.server.keystore.keypassword</name>
          <value>********</value>
        </property>
      

   2. (TLS only) Edit the /etc/hadoop/conf.rollback.secondarynamenode/ssl-server.xml file and remove the hadoop.security.credential.provider.path property.
   3. Log in to the Secondary NameNode host and run the following commands:

      rm -rf /dfs/snn/*
      cd /etc/hadoop/conf.rollback.secondarynamenode/
      sudo -u hdfs hdfs --config /etc/hadoop/conf.rollback.secondarynamenode secondarynamenode -format
      

6. Restart the HDFS service. Open the Cloudera Manager Admin Console, go to the HDFS service page, and select Actions > Restart.

   The Restart Command page displays the progress of the restart. Wait for the page to display the Successfully restarted service message before continuing.

Restart the Key Management Server. Open the Cloudera Manager Admin Console, go to the KMS service page, and select Actions > Start.

Restart the HBase Service. Open the Cloudera Manager Admin Console, go to the HBase service page, and select Actions > Start.

The steps for backing up and restoring databases differ depending on the database vendor and version you select for your cluster and are beyond the scope of this document.

1. Log in to the Cloudera Manager Admin Console.
2. Go to the Sentry service.
3. Click Actions > Start.

1. Restore the file, app.reg, from your backup:
   • Parcel installations

     rm -rf /opt/cloudera/parcels/CDH/lib/hue/app.reg
     cp -rp app.reg_cm5_cdh5_backup /opt/cloudera/parcels/CDH/lib/hue/app.reg

   • Package Installations

     rm -rf /usr/lib/hue/app.reg
     cp -rp app.reg_cm5_cdh5_backup /usr/lib/hue/app.reg

A CDH 6 cluster that is running Kafka can be rolled back to the previous CDH5/CDK versions as long as theinter.broker.protocol.version and log.message.format.version properties have not been set to the new version or removed from the configuration.

1. On the Home > Status tab, click to the right of the cluster name and select Deploy Client Configuration.
2. Click Deploy Client Configuration.

1. On the Home > Status tab, click to the right of the cluster name and select Restart.
2. Click Restart that appears in the next screen to confirm. If you have enabled high availability for HDFS, you can choose Rolling Restart instead to minimize cluster downtime. The Command Details window shows the progress of stopping services.

   When All services successfully started appears, the task is complete and you can close the Command Details window.