Cloudera Docs

Manual upgrade to Cloudera Private Cloud Base

Manual steps to follow for upgrading a CDH or Cloudera Runtime cluster to a higher version of Cloudera Runtime if the Upgrade Wizard fails.

All steps below assume the starting CDH version is at least 5.13.0 or the starting Cloudera Runtime version is at least 7.0.3, because those are the lowest versions that Cloudera Manager 7.1 supports.

The steps below should be executed roughly in the order that they are listed, and should only be executed if the service is configured.

Upgrade Ranger database and apply patches

Required for the following upgrades:

  • CDH 7.0.x to Cloudera Runtime 7.1.1 or higher

  1. Go to the RANGER service.

  2. Select Actions > Upgrade Ranger Database and apply patches and click Upgrade Ranger Database and apply patches to confirm.

Setup Ranger Admin Component

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the Ranger service.

  2. Select Actions > Setup Ranger Admin Component and click Setup Ranger Admin Component to confirm.

Start Ranger

Required for the following upgrades:

  • CDH and 7.0.x to Cloudera Runtime 7.1.1 or higher

  1. Go to the Ranger service.

  2. Select Actions > Start.

Set up the Ranger Plugin service

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the Ranger service.

  2. Select Actions > Setup Ranger Plugin Service and click Setup Ranger Plugin Service to confirm.

Start Kudu

Required for the following upgrades:

  • CDH and 7.0.x to Cloudera Runtime 7.1.1 or higher
  1. Go to the KUDU service.
  2. Select Actions > Start.

Start ZooKeeper

Required for the following upgrades:

  • CDH and 7.0.x to Cloudera Runtime 7.1.1 to 6.0.0 or higher
  1. Go to the ZooKeeper service.
  2. Select Actions > Start.

Upgrade HDFS Metadata

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher
  1. Go to the HDFS service.
  2. Select Actions > Upgrade HDFS Metadata and click Upgrade HDFS Metadata to confirm.

Start HDFS

Required for the following upgrades:

  • CDH and Cloudera Runtime 7.0.x to 7.1.1 or higher
  1. Go to the HDFS service.
  2. Select Actions > Start.

Start YARN QueueManager

Required for the following upgrades:

  • Cloudera Runtime 7.0.x to Cloudera Runtime 7.1.1 or higher

  1. Go to the QueueManager service.

  2. Select Actions > Start.

Import Sentry Polices to Ranger

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the HDFS service.

  2. Select Actions > Import Sentry Policies into Ranger and click Import Sentry Policies into Ranger to confirm.

Start HBASE

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher
  1. Go to the HBASE service.
  2. Select Actions > Start.

Start YARN QueueManager

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the QueueManager service.

  2. Select Actions > Start.

Clean NodeManager Recovery Directory (YARN)

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the YARN service.

  2. Select Actions > Clean NodeManager Recovery Directory and click Clean NodeManager Recovery Directory to confirm.

Reset ACLs on YARN Zookeeper nodes

Required for the following upgrades:

  • Upgrading from CDH to 7.1.1 or higher
  • Any other upgrade if Enable ResourceManager Recovery is enabled for a Resource Manager group (for example, ResourceManager Default Group) and ZooKeeper is a dependency of YARN. Note that when YARN is running in High Availability mode, ResourceManager recovery is always enabled.
  1. Go to the YARN service.
  2. Select Actions > Reset ACLs on YARN Zookeeper nodes
  3. Click Reset ACLs on YARN Zookeeper nodes to confirm.

Install YARN MapReduce Framework Jars

Required for all CDH upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher
  1. Go to the YARN service.
  2. Select Actions > Install YARN MapReduce Framework JARs and click Install YARN MapReduce Framework JARs to confirm.

Start YARN

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher
  1. Go to the YARN service.
  2. Select Actions > Start.

Deploy Client Configuration Files

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. On the Home page, click to the right of the cluster name and select Deploy Client Configuration.
  2. Click the Deploy Client Configuration button in the confirmation pop-up that appears.

Reinitialize Solr State for Upgrade

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the SOLR service.

  2. Select Actions > Reinitialize Solr State for Upgrade and click Reinitialize Solr State for Upgrade to confirm.

Bootstrap Solr Configuration

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the SOLR service.

  2. Select Actions > Bootstrap Solr Configuration and click Bootstrap Solr Configuration to confirm.

Start Solr

Required for the following upgrades:

  • CDH and Cloudera Runtime 7.0.x to Cloudera Runtime 7.1.1 or higher

  1. Go to the SOLR service.

  2. Select Actions > Start.

Bootstrap Solr Collections

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the SOLR service.

  2. Select Actions > Bootstrap Solr Collections and click Bootstrap Solr Collections to confirm.

Create HDFS Home directory

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the infrastructure SOLR service.

  2. Select Actions > Create HDFS Home Dir and click Create HDFS Home Dir to confirm.

Create Ranger Plugin Audit Directory

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the Solr service.

  2. Select Actions > Create Ranger Plugin Audit Directory and click Create Ranger Plugin Audit Directory to confirm.

Start infrastructure Solr

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the infrastructure SOLR service.
  2. Select Actions > Start.

Start HBASE

Required for the following upgrades:

  • Cloudera Runtime 7.0.x to Cloudera Runtime 7.1.1 or higher
  1. Go to the HBASE service.
  2. Select Actions > Start.

Start KAFKA

Required for the following upgrades:

  • CDH and Cloudera Runtime 7.0.x to Cloudera Runtime 7.1.1 or higher
  1. Go to the KAFKA service.
  2. Select Actions > Start.

Create Ranger Kafka Plugin Audit Directory

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the KAFKA service.

  2. Select Actions > Create Ranger Kafka Plugin Audit Directory and click Create Ranger Kafka Plugin Audit Directory to confirm.

Create HBase tables for Atlas

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the ATLAS service.

  2. Select Actions > Create HBase tables for Atlas and click Create HBase tables for Atlas to confirm.

Start Atlas

Required for the following upgrades:

  • CDH and Cloudera Runtime 7.0.x to Cloudera Runtime 7.1.1 or higher

  1. Go to the ATLAS service.

  2. Select Actions > Start.

Create Ranger Atlas Plugin Audit Directory

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the ATLAS service.

  2. Select Actions > Create Ranger Atlas Plugin Audit Directory and click Create Ranger Atlas Plugin Audit Directory to confirm.

Start Phoenix

Required for the following upgrades:

  • CDH Cloudera Runtime 7.0.x to Cloudera Runtime 7.1.1 or higher

  1. Go to the PHOENIX service.

  2. Select Actions > Start.

Install MapReduce Framework Jars

Required for the following upgrades:

  • Cloudera Runtime 7.0.x to Cloudera Runtime 7.1.1 or higher
  1. Go to the YARN service.
  2. Select Actions > Install YARN MapReduce Framework JARs and click Install YARN MapReduce Framework JARs to confirm.

Start YARN

Required for the following upgrades:

  • Cloudera Runtime 7.0.x to Cloudera Runtime 7.1.1 or higher
  1. Go to the YARN service.
  2. Select Actions > Start.

Deploy Client Configuration Files

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. On the Home page, click to the right of the cluster name and select Deploy Client Configuration.
  2. Click the Deploy Client Configuration button in the confirmation pop-up that appears.

Upgrade the Hive Metastore Database

Required for the following upgrades:

  • CDH to 6.0.0 or higher
  1. Go to the Hive service.
  2. If the Hive service is running, stop it:
    1. Select Actions > Stop and click Stop to confirm.
  3. Select Actions > Upgrade Hive Metastore Database Schema and click Upgrade Hive Metastore Database Schema to confirm.
  4. If you have multiple instances of Hive, perform the upgrade on each metastore database.
  5. Select Actions > Validate Hive Metastore Schema and click Validate Hive Metastore Schema to check that the schema is now valid.

Start Hive

Required for the following upgrades:

  • 5.x and Cloudera Runtime 7.0.x to Cloudera Runtime 7.1.1 or higher
  1. Go to the Hive service.
  2. Select Actions > Start.

Create Hive Warehouse Directory

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the HIVE service.

  2. Select Actions > Create Hive Warehouse Directory and click Create Hive Warehouse Directory to confirm.

Create Hive Warehouse External Directory

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the HIVE service.

  2. Select Actions > Create Hive Warehouse External Directory and click Create Hive Warehouse External Directory to confirm.

Create Hive Sys database

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the HIVE service.

  2. Select Actions > Create Hive Sys database and click Create Hive Sys database to confirm.

Create Ranger Plugin Audit Directory

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the HIVE service.

  2. Select Actions > Create Ranger Plugin Audit Directory and click Create Ranger Plugin Audit Directory to confirm.

Start Impala

Required for the following upgrades:

  • CDH to 6.0.0 or higher
  1. Go to the Impala service.
  2. Select Actions > Start.

Create Ranger Plugin Audit Directory

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the Impala service.

  2. Select Actions > Create Ranger Plugin Audit Directory and click Create Ranger Plugin Audit Directory to confirm.

Create Spark Driver Log Dir

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the SPARK_ON_YARN service.

  2. Select Actions > Create Spark Driver Log Dir and click Create Spark Driver Log Dir to confirm.

Start Spark

Required for the following upgrades:

  • CDH and Cloudera Runtime 7.0.x to Cloudera Runtime 7.1.1 or higher

  1. Go to the SPARK_ON_YARN service.

  2. Select Actions > Start.

Start Livy

Required for the following upgrades:

  • Cloudera Runtime 7.0.x to Cloudera Runtime 7.1.1 or higher

  1. Go to the LIVY service.

  2. Select Actions > Start.

Upgrade Oozie Database Schema

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the OOZIE service.

  2. If the OOZIE service is running, stop it:

    Select Actions > Stop and click Stop to confirm.

  3. Select Actions > Upgrade Oozie Database Schema and click Upgrade Oozie Database Schema to confirm.

Updating column types in Oozie Database

If the Oracle database is used for Oozie, then you must update the APP_PATH column type to store values with more than 255 characters. This ensures Oozie does not get stuck in PREP state when your application path exceeds the 255 character limit. If the APP_PATH column type is not updated, then Oozie fails to run the jobs with the following database error message Data too long for column 'app_path'. This scenario is also applicable to coordinator and bundle jobs. For database types other than Oracle, this update is not mandatory for using Oozie. It works without the update if the APP_PATH value does not exceed 255 characters. Also, Oozie's internal database schema validation fails with an unexpected APP_PATH column type. However, this validation does not have any effect. It just logs it's result.

You must update the APP_PATH column type in the WF_JOBS, BUNDLE_JOBS, and COORD_JOBS tables in the Oozie database for the following conditions:
  • When you use the Oracle database for Oozie service.
  • When you are upgrading CDP Runtime from earlier versions to 7.1.9 SP1 CHF1 version or later.

If you do not execute the following statements on the Oozie Oracle database, then the Oozie service fails to run the jobs with a database persistence error. This update is not mandatory, but highly recommended for other database types, such as MySQL, MariaDB, and PostgreSQL, due to the internal database schema validation.

Examples:
  • On the Oozie Oracle database - The following example uses the Oracle sqlplus command-line tool:
    sqlplus <OOZIE_DB_USERNAME>@localhost/<SERVICE_NAME>

SQL> ALTER TABLE <TABLE_NAME> ADD (APP_PATH_TMP CLOB);

Table altered.

SQL> UPDATE <TABLE_NAME> SET APP_PATH_TMP = APP_PATH;

X rows updated.

SQL> ALTER TABLE <TABLE_NAME> DROP COLUMN APP_PATH;

Table altered.

SQL> ALTER TABLE <TABLE_NAME> RENAME COLUMN APP_PATH_TMP TO APP_PATH;

Table altered.
  • On the Oozie MySQL database - The following example uses the MySQL mysql command-line tool:
    $ mysql -u root -p
Enter password:

mysql> use <OOZIE_DATABASE_NAME>;
Database changed

mysql> ALTER TABLE <TABLE_NAME> MODIFY COLUMN app_path text;
Query OK, X rows affected (0.03 sec)
Records: X Duplicates: 0 Warnings: 0

mysql> exit
Bye
  • On the Oozie MariaDB database - The following example uses the MariaDB mysql command-line tool:
    $ mysql -u root -p
Enter password:

MariaDB [(none)]> use <OOZIE_DATABASE_NAME>;
Database changed

MariaDB [OOZIE_DATABASE_NAME]> ALTER TABLE <TABLE_NAME> MODIFY COLUMN app_path text;
Query OK, X rows affected (2.11 sec) 
Records: X Duplicates: 0 Warnings: 0

MariaDB [OOZIE_DATABASE_NAME]> exit
Bye
  • On the Oozie PostgreSQL database - The following example uses the PostgreSQL psql command-line tool:
    $ psql -U postgres
Password for user postgres: *****

postgres=# \c <OOZIE_DATABASE_NAME>;
You are now connected to database "<OOZIE_DATABASE_NAME>" as user "postgres".

OOZIE_DATABASE_NAME=# ALTER TABLE <TABLE_NAME> ALTER COLUMN app_path type text;
ALTER TABLE

OOZIE_DATABASE_NAME=# \q

Upgrade Oozie SharedLib

Required for the following upgrades:

  • CDH and Cloudera Runtime 7.0.x to Cloudera Runtime 7.1.1 or higher
  1. Go to the Oozie service.
  2. If the OOZIE service is stopped, start it:

    Select Actions > Start and click Start to confirm.

  3. Select Actions > Install Oozie SharedLib and click Install Oozie SharedLib to confirm.

Upload Tez tar file to HDFS

Required for the following upgrades:

  • CDH and Cloudera Runtime 7.0.x to Cloudera Runtime 7.1.1 or higher

  1. Go to the TEZ service.

  2. Select Actions > Upload Tez tar file to HDFS and click Upload Tez tar file to HDFS to confirm.

Migrate Hive tables for CDP upgrade

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the HIVE_ON_TEZ service.

  2. Select Actions > Migrate Hive tables for CDP upgrade and click Migrate Hive tables for CDP upgrade to confirm.

Create Ranger Plugin Audit Directory

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the Hive-on-Tez service.

  2. Select Actions > Create Ranger Plugin Audit Directory and click Create Ranger Plugin Audit Directory to confirm.

Start Hive on Tez

Required for the following upgrades:

  • CDH and Cloudera Runtime 7.0.x to Cloudera Runtime 7.1.1 or higher
  1. Go to the Hive-on-Tez service.
  2. Select Actions > Start.

Start Hue

Required for the following upgrades:

  • CDH and Cloudera Runtime 7.0.x to Cloudera Runtime 7.1.1 or higher

  1. Go to the HUE service.
  2. Select Actions > Start.

Start the Remaining Cluster Services

  1. Use rolling restart or full restart.

  2. Ensure that all services are started or restarted. You can use Cloudera Manager to start the cluster, or you can restart the services individually. The Cloudera Manager Home page indicates which services have stale configurations and require restarting.

  3. To start or restart the cluster:

    1. On the Home > Status page, click the down arrow to the right of the cluster name and select Start or Restart.

    2. Click Start that appears in the next screen to confirm. The Command Details window shows the progress of starting services.

    3. When All services successfully started appears, the task is complete and you can close the Command Details window.

Validate the Hive Metastore Database Schema

Required for the following upgrades:

  • CDH to 6.0.0 or higher
  1. Select Actions > Validate Hive Metastore Schema and click Validate Hive Metastore Schema to confirm.
  2. If you have multiple instances of Hive, perform the validation on each metastore database.
  3. Select Actions > Validate Hive Metastore Schema and click Validate Hive Metastore Schema to check that the schema is now valid.

Test the Cluster and Finalize HDFS Metadata

To determine if you can finalize the upgrade, run important workloads and ensure that they are successful. After you have finalized the upgrade, you cannot roll back to a previous version of HDFS without using backups. Verifying that you are ready to finalize the upgrade can take a long time.

When you are ready to finalize the upgrade, do the following:
    1. Go to the HDFS service.
    2. Click the Instances tab.
    3. Click the link for the NameNode instance. If you have enabled high availability for HDFS, click the link labeled NameNode (Active).

      The NameNode instance page displays.

    4. Select Actions > Finalize Metadata Upgrade and click Finalize Metadata Upgrade to confirm.

Clear the Upgrade State Table

After completing all of the previous steps, do the following to complete the upgrade:
  1. Log in to the Cloudera Manager server host.
  2. Stop the Cloudera Manager Server.
    sudo systemctl stop cloudera-scm-server
  3. Log in to the command-line environment for the Cloudera Manager database. (mysql, sqlplus, or postgres psql).
  4. Run the following command:
    DELETE FROM UPGRADE_STATE;
  5. Start the Cloudera Manager Server.
    sudo systemctl start cloudera-scm-server