CDH 5 to CDP Private Cloud BasePDF version

Manual upgrade to Cloudera Base on premises

Manual steps to follow for upgrading a CDH or Cloudera Runtime cluster to a higher version of Cloudera Runtime if the Upgrade Wizard fails.

All steps below assume the starting CDH version is at least 5.13.0 or the starting Cloudera Runtime version is at least 7.0.3, because those are the lowest versions that Cloudera Manager 7.1 supports.

The steps below should be executed roughly in the order that they are listed, and should only be executed if the service is configured.

Required for the following upgrades:

  • CDH 7.0.x to Cloudera Runtime 7.1.1 or higher

  1. Go to the RANGER service.

  2. Select Actions > Upgrade Ranger Database and apply patches and click Upgrade Ranger Database and apply patches to confirm.

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the Ranger service.

  2. Select Actions > Setup Ranger Admin Component and click Setup Ranger Admin Component to confirm.

Required for the following upgrades:

  • CDH and 7.0.x to Cloudera Runtime 7.1.1 or higher

  1. Go to the Ranger service.

  2. Select Actions > Start.

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the Ranger service.

  2. Select Actions > Setup Ranger Plugin Service and click Setup Ranger Plugin Service to confirm.

Required for the following upgrades:

  • CDH and 7.0.x to Cloudera Runtime 7.1.1 or higher
  1. Go to the KUDU service.
  2. Select Actions > Start.

Required for the following upgrades:

  • CDH and 7.0.x to Cloudera Runtime 7.1.1 to 6.0.0 or higher
  1. Go to the ZooKeeper service.
  2. Select Actions > Start.

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher
  1. Go to the HDFS service.
  2. Select Actions > Upgrade HDFS Metadata and click Upgrade HDFS Metadata to confirm.

Required for the following upgrades:

  • CDH and Cloudera Runtime 7.0.x to 7.1.1 or higher
  1. Go to the HDFS service.
  2. Select Actions > Start.

Required for the following upgrades:

  • Cloudera Runtime 7.0.x to Cloudera Runtime 7.1.1 or higher

  1. Go to the QueueManager service.

  2. Select Actions > Start.

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the HDFS service.

  2. Select Actions > Import Sentry Policies into Ranger and click Import Sentry Policies into Ranger to confirm.

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher
  1. Go to the HBASE service.
  2. Select Actions > Start.

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the QueueManager service.

  2. Select Actions > Start.

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the YARN service.

  2. Select Actions > Clean NodeManager Recovery Directory and click Clean NodeManager Recovery Directory to confirm.

Required for the following upgrades:

  • Upgrading from CDH to Cloudera Runtime 7.1.1 or higher
  • Any other upgrade if Enable ResourceManager Recovery is enabled for a Resource Manager group (for example, ResourceManager Default Group) and ZooKeeper is a dependency of YARN. Note that when YARN is running in High Availability mode, ResourceManager recovery is always enabled.
  1. Go to the YARN service.
  2. Select Actions > Reset ACLs on YARN Zookeeper nodes
  3. Click Reset ACLs on YARN Zookeeper nodes to confirm.

Required for all CDH upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher
  1. Go to the YARN service.
  2. Select Actions > Install YARN MapReduce Framework JARs and click Install YARN MapReduce Framework JARs to confirm.

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher
  1. Go to the YARN service.
  2. Select Actions > Start.

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. On the Home page, click to the right of the cluster name and select Deploy Client Configuration.
  2. Click the Deploy Client Configuration button in the confirmation pop-up that appears.

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the SOLR service.

  2. Select Actions > Reinitialize Solr State for Upgrade and click Reinitialize Solr State for Upgrade to confirm.

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the SOLR service.

  2. Select Actions > Bootstrap Solr Configuration and click Bootstrap Solr Configuration to confirm.

Required for the following upgrades:

  • CDH and Cloudera Runtime 7.0.x to Cloudera Runtime 7.1.1 or higher

  1. Go to the SOLR service.

  2. Select Actions > Start.

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the SOLR service.

  2. Select Actions > Bootstrap Solr Collections and click Bootstrap Solr Collections to confirm.

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the infrastructure SOLR service.

  2. Select Actions > Create HDFS Home Dir and click Create HDFS Home Dir to confirm.

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the Solr service.

  2. Select Actions > Create Ranger Plugin Audit Directory and click Create Ranger Plugin Audit Directory to confirm.

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the infrastructure SOLR service.
  2. Select Actions > Start.

Required for the following upgrades:

  • Cloudera Runtime 7.0.x to Cloudera Runtime 7.1.1 or higher
  1. Go to the HBASE service.
  2. Select Actions > Start.

Required for the following upgrades:

  • CDH and Cloudera Runtime 7.0.x to Cloudera Runtime 7.1.1 or higher
  1. Go to the KAFKA service.
  2. Select Actions > Start.

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the KAFKA service.

  2. Select Actions > Create Ranger Kafka Plugin Audit Directory and click Create Ranger Kafka Plugin Audit Directory to confirm.

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the ATLAS service.

  2. Select Actions > Create HBase tables for Atlas and click Create HBase tables for Atlas to confirm.

Required for the following upgrades:

  • CDH and Cloudera Runtime 7.0.x to Cloudera Runtime 7.1.1 or higher

  1. Go to the ATLAS service.

  2. Select Actions > Start.

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the ATLAS service.

  2. Select Actions > Create Ranger Atlas Plugin Audit Directory and click Create Ranger Atlas Plugin Audit Directory to confirm.

Required for the following upgrades:

  • CDH Cloudera Runtime 7.0.x to Cloudera Runtime 7.1.1 or higher

  1. Go to the PHOENIX service.

  2. Select Actions > Start.

Required for the following upgrades:

  • Cloudera Runtime 7.0.x to Cloudera Runtime 7.1.1 or higher
  1. Go to the YARN service.
  2. Select Actions > Install YARN MapReduce Framework JARs and click Install YARN MapReduce Framework JARs to confirm.

Required for the following upgrades:

  • Cloudera Runtime 7.0.x to Cloudera Runtime 7.1.1 or higher
  1. Go to the YARN service.
  2. Select Actions > Start.

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. On the Home page, click to the right of the cluster name and select Deploy Client Configuration.
  2. Click the Deploy Client Configuration button in the confirmation pop-up that appears.

Required for the following upgrades:

  • CDH to 6.0.0 or higher
  1. Go to the Hive service.
  2. If the Hive service is running, stop it:
    1. Select Actions > Stop and click Stop to confirm.
  3. Select Actions > Upgrade Hive Metastore Database Schema and click Upgrade Hive Metastore Database Schema to confirm.
  4. If you have multiple instances of Hive, perform the upgrade on each metastore database.
  5. Select Actions > Validate Hive Metastore Schema and click Validate Hive Metastore Schema to check that the schema is now valid.

Required for the following upgrades:

  • 5.x and Cloudera Runtime 7.0.x to Cloudera Runtime 7.1.1 or higher
  1. Go to the Hive service.
  2. Select Actions > Start.

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the HIVE service.

  2. Select Actions > Create Hive Warehouse Directory and click Create Hive Warehouse Directory to confirm.

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the HIVE service.

  2. Select Actions > Create Hive Warehouse External Directory and click Create Hive Warehouse External Directory to confirm.

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the HIVE service.

  2. Select Actions > Create Hive Sys database and click Create Hive Sys database to confirm.

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the HIVE service.

  2. Select Actions > Create Ranger Plugin Audit Directory and click Create Ranger Plugin Audit Directory to confirm.

Required for the following upgrades:

  • CDH to 6.0.0 or higher
  1. Go to the Impala service.
  2. Select Actions > Start.

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the Impala service.

  2. Select Actions > Create Ranger Plugin Audit Directory and click Create Ranger Plugin Audit Directory to confirm.

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the SPARK_ON_YARN service.

  2. Select Actions > Create Spark Driver Log Dir and click Create Spark Driver Log Dir to confirm.

Required for the following upgrades:

  • CDH and Cloudera Runtime 7.0.x to Cloudera Runtime 7.1.1 or higher

  1. Go to the SPARK_ON_YARN service.

  2. Select Actions > Start.

Required for the following upgrades:

  • Cloudera Runtime 7.0.x to Cloudera Runtime 7.1.1 or higher

  1. Go to the LIVY service.

  2. Select Actions > Start.

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the OOZIE service.

  2. If the OOZIE service is running, stop it:

    Select Actions > Stop and click Stop to confirm.

  3. Select Actions > Upgrade Oozie Database Schema and click Upgrade Oozie Database Schema to confirm.

If the Oracle database is used for Oozie, then you must update the APP_PATH column type to store values with more than 255 characters. This ensures Oozie does not get stuck in PREP state when your application path exceeds the 255 character limit. If the APP_PATH column type is not updated, then Oozie fails to run the jobs with the following database error message Data too long for column 'app_path'. This scenario is also applicable to coordinator and bundle jobs. For database types other than Oracle, this update is not mandatory for using Oozie. It works without the update if the APP_PATH value does not exceed 255 characters. Also, Oozie's internal database schema validation fails with an unexpected APP_PATH column type. However, this validation does not have any effect. It just logs it's result.

You must update the APP_PATH column type in the WF_JOBS, BUNDLE_JOBS, and COORD_JOBS tables in the Oozie database for the following conditions:
  • When you use the Oracle database for Oozie service.
  • When you are upgrading Cloudera Runtime from earlier versions to Cloudera Base on premises 7.1.9 SP1 CHF1 version or later.

If you do not execute the following statements on the Oozie Oracle database, then the Oozie service fails to run the jobs with a database persistence error. This update is not mandatory, but highly recommended for other database types, such as MySQL, MariaDB, and PostgreSQL, due to the internal database schema validation.

Examples:
  • On the Oozie Oracle database - The following example uses the Oracle sqlplus command-line tool:
    sqlplus <OOZIE_DB_USERNAME>@localhost/<SERVICE_NAME>

SQL> ALTER TABLE <TABLE_NAME> ADD (APP_PATH_TMP CLOB);

Table altered.

SQL> UPDATE <TABLE_NAME> SET APP_PATH_TMP = APP_PATH;

X rows updated.

SQL> ALTER TABLE <TABLE_NAME> DROP COLUMN APP_PATH;

Table altered.

SQL> ALTER TABLE <TABLE_NAME> RENAME COLUMN APP_PATH_TMP TO APP_PATH;

Table altered.
  • On the Oozie MySQL database - The following example uses the MySQL mysql command-line tool:
    $ mysql -u root -p
Enter password:

mysql> use <OOZIE_DATABASE_NAME>;
Database changed

mysql> ALTER TABLE <TABLE_NAME> MODIFY COLUMN app_path text;
Query OK, X rows affected (0.03 sec)
Records: X Duplicates: 0 Warnings: 0

mysql> exit
Bye
  • On the Oozie MariaDB database - The following example uses the MariaDB mysql command-line tool:
    $ mysql -u root -p
Enter password:

MariaDB [(none)]> use <OOZIE_DATABASE_NAME>;
Database changed

MariaDB [OOZIE_DATABASE_NAME]> ALTER TABLE <TABLE_NAME> MODIFY COLUMN app_path text;
Query OK, X rows affected (2.11 sec) 
Records: X Duplicates: 0 Warnings: 0

MariaDB [OOZIE_DATABASE_NAME]> exit
Bye
  • On the Oozie PostgreSQL database - The following example uses the PostgreSQL psql command-line tool:
    $ psql -U postgres
Password for user postgres: *****

postgres=# \c <OOZIE_DATABASE_NAME>;
You are now connected to database "<OOZIE_DATABASE_NAME>" as user "postgres".

OOZIE_DATABASE_NAME=# ALTER TABLE <TABLE_NAME> ALTER COLUMN app_path type text;
ALTER TABLE

OOZIE_DATABASE_NAME=# \q

Required for the following upgrades:

  • CDH and Cloudera Runtime 7.0.x to Cloudera Runtime 7.1.1 or higher
  1. Go to the Oozie service.
  2. If the OOZIE service is stopped, start it:

    Select Actions > Start and click Start to confirm.

  3. Select Actions > Install Oozie SharedLib and click Install Oozie SharedLib to confirm.

Required for the following upgrades:

  • CDH and Cloudera Runtime 7.0.x to Cloudera Runtime 7.1.1 or higher

  1. Go to the TEZ service.

  2. Select Actions > Upload Tez tar file to HDFS and click Upload Tez tar file to HDFS to confirm.

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the HIVE_ON_TEZ service.

  2. Select Actions > Migrate Hive tables for CDP upgrade and click Migrate Hive tables for CDP upgrade to confirm.

Required for the following upgrades:

  • CDH to Cloudera Runtime 7.1.1 or higher

  1. Go to the Hive-on-Tez service.

  2. Select Actions > Create Ranger Plugin Audit Directory and click Create Ranger Plugin Audit Directory to confirm.

Required for the following upgrades:

  • CDH and Cloudera Runtime 7.0.x to Cloudera Runtime 7.1.1 or higher
  1. Go to the Hive-on-Tez service.
  2. Select Actions > Start.

Required for the following upgrades:

  • CDH and Cloudera Runtime 7.0.x to Cloudera Runtime 7.1.1 or higher

  1. Go to the HUE service.
  2. Select Actions > Start.

  1. Use rolling restart or full restart.

  2. Ensure that all services are started or restarted. You can use Cloudera Manager to start the cluster, or you can restart the services individually. The Cloudera Manager Home page indicates which services have stale configurations and require restarting.

  3. To start or restart the cluster:

    1. On the Home > Status page, click the down arrow to the right of the cluster name and select Start or Restart.

    2. Click Start that appears in the next screen to confirm. The Command Details window shows the progress of starting services.

    3. When All services successfully started appears, the task is complete and you can close the Command Details window.

Required for the following upgrades:

  • CDH to 6.0.0 or higher
  1. Select Actions > Validate Hive Metastore Schema and click Validate Hive Metastore Schema to confirm.
  2. If you have multiple instances of Hive, perform the validation on each metastore database.
  3. Select Actions > Validate Hive Metastore Schema and click Validate Hive Metastore Schema to check that the schema is now valid.

To determine if you can finalize the upgrade, run important workloads and ensure that they are successful. After you have finalized the upgrade, you cannot roll back to a previous version of HDFS without using backups. Verifying that you are ready to finalize the upgrade can take a long time.

When you are ready to finalize the upgrade, do the following:
    1. Go to the HDFS service.
    2. Click the Instances tab.
    3. Click the link for the NameNode instance. If you have enabled high availability for HDFS, click the link labeled NameNode (Active).

      The NameNode instance page displays.

    4. Select Actions > Finalize Metadata Upgrade and click Finalize Metadata Upgrade to confirm.

After completing all of the previous steps, do the following to complete the upgrade:
  1. Log in to the Cloudera Manager server host.
  2. Stop the Server.
    sudo systemctl stop cloudera-scm-server
  3. Log in to the command-line environment for the Cloudera Manager database. (mysql, sqlplus, or postgres psql).
  4. Run the following command:
    DELETE FROM UPGRADE_STATE;
  5. Start the Server.
    sudo systemctl start cloudera-scm-server