Rolling back a Cloudera Private Cloud Base 7 upgrade to CDH 5

To ensure successful rollback, you must disable Auto-TLS configuration before you rollback to Cloudera Manager 5.16.2 from Cloudera Manager 7.x.

1. When you perform rollback to Cloudera Manager 5.16.2 (with Auto-TLS enabled) from Cloudera Manager 7.x, ensure you disable Auto-TLS configuration in Cloudera Manager 7.x. For the instructions about disabling Auto-TLS, see the KB article.
2. While performing rollback to Cloudera Manager 5.16.2 (without Auto-TLS enabled) from Cloudera Manager 7.x, and if you have manually enabled Auto-TLS post upgrade, then perform the following steps to disable Auto-TLS configuration in Cloudera Manager 7.x:
   1. Log into Cloudera Manager as an Administrator.
   2. Go to Support > API Explorer.

   3. Locate and click the /clusters/{clusterName}/commands/disableTls endpoint for ClustersResource API to view the API parameters.

   4. Click Try it out.
   5. Enter the name of your cluster in the clusterName field.
   6. Click Execute.
   7. Go to Adminisration > Settings > Security and deselect the following:
      • Use TLS Encryption for Admin Console
      • Use TLS Encryption for Agents
      • Use TLS Authentication of Agents to Server
      • Verify Agent Hostname Against Certificate
   8. Ensure that the following fields are empty:

      • Host certificate generator command
      • Server SSL Certificate Host Name
   9. Set the use_tls=0 in the config.ini file of the Cloudera Manager agent.
   10. Ensure to restore the Load Balancer's configuration for Impala and Oozie as shown in the below examples:
       1. Remove the SSL parameters from the /etc/haproxy/haproxy.cfg configuration file on port 25003 to avoid impala-shell to connect with SSL:

          /etc/haproxy/haproxy.cfg
          ...
          ...
          #---------------------------------------------------------------------
          # main frontend which proxys to the backends
          #---------------------------------------------------------------------
          frontend impala_front
            bind            *:25003 ssl crt /var/lib/cloudera-scm-agent/agent-cert/cdep-host_key_cert_chain_decrypted.pem
            mode            tcp
            option           tcplog
            default_backend       impala
          ...
          ...

          The /etc/haproxy/haproxy.cfg configuration file without SSL parameters should look as shown below:

          /etc/haproxy/haproxy.cfg
          ...
          ...
          #---------------------------------------------------------------------
          # main frontend which proxys to the backends
          #---------------------------------------------------------------------
          frontend impala_front
            bind            *:25003
            mode            tcp
            option           tcplog
            default_backend       impala
          ...
          ...

       2. On the Oozie load balancer host, set the load balancer ports to 5002 for HTTPS and 5000 for HTTP in the /etc/haproxy/haproxy.cfg configuration file as shown below:

          #---------------------------------------------------------------------
          # main frontend which proxys to the backends
          #---------------------------------------------------------------------
          frontend                        oozie_front
              bind                        *:5002 ssl crt /var/lib/cloudera-scm-agent/agent-cert/cdep-host_key_cert_chain_decrypted.pem
              default_backend             oozie
          #---------------------------------------------------------------------
          # round robin balancing between the various backends
          #---------------------------------------------------------------------
          backend oozie
              balance                     roundrobin
              server  oozie1 quasar-ebrvlp-3.quasar-ebrvlp.root.hwx.site:11443/oozie check ssl ca-file /var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_cacerts.pem
              server  oozie2 quasar-ebrvlp-1.quasar-ebrvlp.root.hwx.site:11443/oozie check ssl ca-file /var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_cacerts.pem
              server  oozie3 quasar-ebrvlp-2.quasar-ebrvlp.root.hwx.site:11443/oozie check ssl ca-file /var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_cacerts.pem
          #---------------------------------------------------------------------
          # main frontend which proxys to the http backends
          #---------------------------------------------------------------------
          frontend                        oozie_front_http
              bind                        *:5000
              default_backend             oozie_http
          #---------------------------------------------------------------------
          # round robin balancing between the various http backends
          #---------------------------------------------------------------------
          backend oozie_http
              balance                     roundrobin
              server  oozie_http1 quasar-ebrvlp-3.quasar-ebrvlp.root.hwx.site:11000/oozie check
              server  oozie_http2 quasar-ebrvlp-1.quasar-ebrvlp.root.hwx.site:11000/oozie check
              server  oozie_http3 quasar-ebrvlp-2.quasar-ebrvlp.root.hwx.site:11000/oozie check 

       3. After the rollback, run the following command to restart the HAProxy service as a root user:

          service haproxy restart

1. On the Home > Status tab, click the Actions menu and select Stop.
2. Click Stop in the confirmation screen. The Command Details window shows the progress of stopping services.

   When All services successfully stopped appears, the task is complete and you can close the Command Details window.

3. Go to the YARN service and click Actions > Clean NodeManager Recovery Directory. The CDH 5 NodeManager will not start up after the downgrade if it finds CDP 7.x data in the recovery directory. The format and content of the NodeManager's recovery state store was changed between CDH 5.x and CDP 7.x. The recovery directory used by CDP 7.x must be cleaned up as part of the downgrade to CDH 5.

Follow these steps only if your cluster was upgraded using Cloudera parcels.

1. Log in to the Cloudera Manager Admin Console.
2. Select Hosts > Parcels.

   A list of parcels displays.

3. Locate the CDH 5 parcel and click Activate. (This automatically deactivates the Cloudera Private Cloud Base 7 parcel.) See Activating a Parcel for more information. If the parcel is not available, use the Download button to download the parcel.
4. If you include any additional components in your cluster, such as Search or Impala, click Activate for those parcels.

1. Stop the Cloudera Management Service.
   1. Log in to the Cloudera Manager Admin Console.
   2. Select Clusters > Cloudera Management Service.
   3. Select Actions > Stop.
2. Stop the Cloudera Manager Server.

   sudo systemctl stop cloudera-scm-server

3. Hard stop the Cloudera Manager agents. Run the following command on all hosts:

   sudo systemctl stop cloudera-scm-supervisord.service
   

Follow these steps only if your cluster was upgraded using packages.

Use the backup of CDH that was taken before the upgrade to restore Cloudera Manager Server files and directories. Substitute the path to your backup directory for cm7_cdh5 in the following steps:

1. On the host where the Event Server role is configured to run, restore the Events Server directory from the CM 7/CDH 5 backup.

   rm -rf /var/lib/cloudera-scm-eventserver/*
   cp -rp /var/lib/cloudera-scm-eventserver_cm7_cdh5/* /var/lib/cloudera-scm-eventserver/

2. Remove the Agent runtime state. Run the following command on all hosts:

   rm -rf /var/run/cloudera-scm-agent /var/lib/cloudera-scm-agent/response.avro

   This command may return a message similar to: rm: cannot remove ‘/var/run/cloudera-scm-agent/process’: Device or resource busy. You can ignore this message.

3. On the host where the Service Monitor is running, restore the Service Monitor directory:

   rm -rf /var/lib/cloudera-service-monitor/*
   cp -rp /var/lib/cloudera-service-monitor_cm7_cdh5/* /var/lib/cloudera-service-monitor/

4. On the host where the Host Monitor is running, restore the Host Monitor directory:

   rm -rf /var/lib/cloudera-host-monitor/*
   cp -rp /var/lib/cloudera-host-monitor_cm7_cdh5/* /var/lib/cloudera-host-monitor/

5. Restore the Cloudera Navigator storage directory from the CM 7/CDH 5 backup.

   rm -rf /var/lib/cloudera-scm-navigator/*
   cp -rp /var/lib/cloudera-scm-navigator_cm7_cdh5/* /var/lib/cloudera-scm-navigator/

At this point, roll-backing Cloudera Manager is not required and is completely optional. But, if you want to rollback Cloudera Manager as well, follow steps as discussed in (Optional) Cloudera Manager Rollback Steps prior to going to the next step which is Start Cloudera Manager.

1. Log in to the Cloudera Manager server host.
2. Start the Cloudera Manager Server.

   sudo systemctl start cloudera-scm-server

3. Start the Cloudera Manager Agent.

   Run the following commands on all cluster hosts:

   sudo systemctl start cloudera-scm-agent

4. Start the Cloudera Management Service.
   1. Log in to the Cloudera Manager Admin Console.
   2. Select Clusters > Cloudera Management Service.
   3. Select Actions > Start.
5. Stop the cluster. In the Cloudera Manager Admin Console, click the Actions menu for the cluster and select Stop.

The steps for backing up and restoring databases differ depending on the database vendor and version you select for your cluster and are beyond the scope of this document.

1. On the Cloudera Manager Home page, click the Actions menu and select Deploy Client Configuration.
2. Click Deploy Client Configuration.

1. On the Cloudera Manager Home page, click the Actions menu and select Restart.
2. Click Restart that appears in the next screen to confirm. If you have enabled high availability for HDFS, you can choose Rolling Restart instead to minimize cluster downtime. The Command Details window shows the progress of stopping services.

   When All services successfully started appears, the task is complete and you can close the Command Details window.