Cloudera Docs

Manual upgrade to Cloudera Base on premises

Manual steps to follow for upgrading a Cloudera Runtime cluster to a higher version of Cloudera Runtime if the Upgrade Wizard fails.

All the following steps assume the starting Cloudera Runtime version is at least 7.0.3, because those are the lowest versions that Cloudera Manager 7.1 supports. The following steps are required when you are upgrading Cloudera Runtime cluster from Cloudera Runtime 7.0.3 to Cloudera Runtime 7.1.1 or higher versions.

The following steps should be executed roughly in the order that they are listed, and should only be executed if the service is configured.

Upgrade Ranger database and apply patches

  1. Go to the RANGER service.

  2. Select Actions > Upgrade Ranger Database and apply patches and click Upgrade Ranger Database and apply patches to confirm.

Setup Ranger Admin Component

  1. Go to the Ranger service.

  2. Select Actions > Setup Ranger Admin Component and click Setup Ranger Admin Component to confirm.

Start Ranger

  1. Go to the Ranger service.

  2. Select Actions > Start.

Set up the Ranger Plugin service

  1. Go to the Ranger service.

  2. Select Actions > Setup Ranger Plugin Service and click Setup Ranger Plugin Service to confirm.

Start Kudu

  1. Go to the KUDU service.
  2. Select Actions > Start.

Start ZooKeeper

  1. Go to the ZooKeeper service.
  2. Select Actions > Start.

Upgrade HDFS Metadata

  1. Go to the HDFS service.
  2. Select Actions > Upgrade HDFS Metadata and click Upgrade HDFS Metadata to confirm.

Start HDFS

  1. Go to the HDFS service.
  2. Select Actions > Start.

Start YARN QueueManager

  1. Go to the QueueManager service.

  2. Select Actions > Start.

Import Sentry Polices to Ranger

  1. Go to the HDFS service.

  2. Select Actions > Import Sentry Policies into Ranger and click Import Sentry Policies into Ranger to confirm.

Start HBASE

  1. Go to the HBASE service.
  2. Select Actions > Start.

Start YARN QueueManager

  1. Go to the QueueManager service.

  2. Select Actions > Start.

Clean NodeManager Recovery Directory (YARN)

  1. Go to the YARN service.

  2. Select Actions > Clean NodeManager Recovery Directory and click Clean NodeManager Recovery Directory to confirm.

Reset ACLs on YARN Zookeeper nodes

Any other upgrade if Enable ResourceManager Recovery is enabled for a Resource Manager group (for example, ResourceManager Default Group) and ZooKeeper is a dependency of YARN. Note that when YARN is running in High Availability mode, ResourceManager recovery is always enabled.

  1. Go to the YARN service.
  2. Select Actions > Reset ACLs on YARN Zookeeper nodes
  3. Click Reset ACLs on YARN Zookeeper nodes to confirm.

Install YARN MapReduce Framework Jars

  1. Go to the YARN service.
  2. Select Actions > Install YARN MapReduce Framework JARs and click Install YARN MapReduce Framework JARs to confirm.

Start YARN

  1. Go to the YARN service.
  2. Select Actions > Start.

Deploy Client Configuration Files

  1. On the Home page, click to the right of the cluster name and select Deploy Client Configuration.
  2. Click the Deploy Client Configuration button in the confirmation pop-up that appears.

Reinitialize Solr State for Upgrade

  1. Go to the SOLR service.

  2. Select Actions > Reinitialize Solr State for Upgrade and click Reinitialize Solr State for Upgrade to confirm.

Bootstrap Solr Configuration

  1. Go to the SOLR service.

  2. Select Actions > Bootstrap Solr Configuration and click Bootstrap Solr Configuration to confirm.

Start Solr

  1. Go to the SOLR service.

  2. Select Actions > Start.

Bootstrap Solr Collections

  1. Go to the SOLR service.

  2. Select Actions > Bootstrap Solr Collections and click Bootstrap Solr Collections to confirm.

Create HDFS Home directory

  1. Go to the infrastructure SOLR service.

  2. Select Actions > Create HDFS Home Dir and click Create HDFS Home Dir to confirm.

Create Ranger Plugin Audit Directory

  1. Go to the Solr service.

  2. Select Actions > Create Ranger Plugin Audit Directory and click Create Ranger Plugin Audit Directory to confirm.

Start infrastructure Solr

  1. Go to the infrastructure SOLR service.
  2. Select Actions > Start.

Start HBASE

  1. Go to the HBASE service.
  2. Select Actions > Start.

Start KAFKA

  1. Go to the KAFKA service.
  2. Select Actions > Start.

Create Ranger Kafka Plugin Audit Directory

  1. Go to the KAFKA service.

  2. Select Actions > Create Ranger Kafka Plugin Audit Directory and click Create Ranger Kafka Plugin Audit Directory to confirm.

Create HBase tables for Atlas

  1. Go to the ATLAS service.

  2. Select Actions > Create HBase tables for Atlas and click Create HBase tables for Atlas to confirm.

Start Atlas

  1. Go to the ATLAS service.

  2. Select Actions > Start.

Create Ranger Atlas Plugin Audit Directory

  1. Go to the ATLAS service.

  2. Select Actions > Create Ranger Atlas Plugin Audit Directory and click Create Ranger Atlas Plugin Audit Directory to confirm.

Start Phoenix

  1. Go to the PHOENIX service.

  2. Select Actions > Start.

Install MapReduce Framework Jars

  1. Go to the YARN service.
  2. Select Actions > Install YARN MapReduce Framework JARs and click Install YARN MapReduce Framework JARs to confirm.

Start YARN

  1. Go to the YARN service.
  2. Select Actions > Start.

Deploy Client Configuration Files

  1. On the Home page, click to the right of the cluster name and select Deploy Client Configuration.
  2. Click the Deploy Client Configuration button in the confirmation pop-up that appears.

Upgrade the Hive Metastore Database

  1. Go to the Hive service.
  2. If the Hive service is running, stop it:
    1. Select Actions > Stop and click Stop to confirm.
  3. Select Actions > Upgrade Hive Metastore Database Schema and click Upgrade Hive Metastore Database Schema to confirm.
  4. If you have multiple instances of Hive, perform the upgrade on each metastore database.
  5. Select Actions > Validate Hive Metastore Schema and click Validate Hive Metastore Schema to check that the schema is now valid.

Start Hive

  1. Go to the Hive service.
  2. Select Actions > Start.

Create Hive Warehouse Directory

  1. Go to the HIVE service.

  2. Select Actions > Create Hive Warehouse Directory and click Create Hive Warehouse Directory to confirm.

Create Hive Warehouse External Directory

  1. Go to the HIVE service.

  2. Select Actions > Create Hive Warehouse External Directory and click Create Hive Warehouse External Directory to confirm.

Create Hive Sys database

  1. Go to the HIVE service.

  2. Select Actions > Create Hive Sys database and click Create Hive Sys database to confirm.

Create Ranger Plugin Audit Directory

  1. Go to the HIVE service.

  2. Select Actions > Create Ranger Plugin Audit Directory and click Create Ranger Plugin Audit Directory to confirm.

Start Impala

  1. Go to the Impala service.
  2. Select Actions > Start.

Create Ranger Plugin Audit Directory

  1. Go to the Impala service.

  2. Select Actions > Create Ranger Plugin Audit Directory and click Create Ranger Plugin Audit Directory to confirm.

Create Spark Driver Log Dir

  1. Go to the SPARK_ON_YARN service.

  2. Select Actions > Create Spark Driver Log Dir and click Create Spark Driver Log Dir to confirm.

Start Spark

  1. Go to the SPARK_ON_YARN service.

  2. Select Actions > Start.

Start Livy

  1. Go to the LIVY service.

  2. Select Actions > Start.

Upgrade Oozie Database Schema

  1. Go to the OOZIE service.

  2. If the OOZIE service is running, stop it:

    Select Actions > Stop and click Stop to confirm.

  3. Select Actions > Upgrade Oozie Database Schema and click Upgrade Oozie Database Schema to confirm.

Updating column types in Oozie Database

If the Oracle database is used for Oozie, then you must update the APP_PATH column type to store values with more than 255 characters. This ensures Oozie does not get stuck in PREP state when your application path exceeds the 255 character limit. If the APP_PATH column type is not updated, then Oozie fails to run the jobs with the following database error message Data too long for column 'app_path'. This scenario is also applicable to coordinator and bundle jobs. For database types other than Oracle, this update is not mandatory for using Oozie. It works without the update if the APP_PATH value does not exceed 255 characters. Also, Oozie's internal database schema validation fails with an unexpected APP_PATH column type. However, this validation does not have any effect. It just logs it's result.

You must update the APP_PATH column type in the WF_JOBS, BUNDLE_JOBS, and COORD_JOBS tables in the Oozie database for the following conditions:
  • When you use the Oracle database for Oozie service.
  • When you are upgrading Cloudera Runtime from earlier versions to Cloudera Base on premises 7.1.9 SP1 CHF1 version or later.

If you do not execute the following statements on the Oozie Oracle database, then the Oozie service fails to run the jobs with a database persistence error. This update is not mandatory, but highly recommended for other database types, such as MySQL, MariaDB, and PostgreSQL, due to the internal database schema validation.

Examples:
  • On the Oozie Oracle database - The following example uses the Oracle sqlplus command-line tool:
    sqlplus <OOZIE_DB_USERNAME>@localhost/<SERVICE_NAME>

SQL> ALTER TABLE <TABLE_NAME> ADD (APP_PATH_TMP CLOB);

Table altered.

SQL> UPDATE <TABLE_NAME> SET APP_PATH_TMP = APP_PATH;

X rows updated.

SQL> ALTER TABLE <TABLE_NAME> DROP COLUMN APP_PATH;

Table altered.

SQL> ALTER TABLE <TABLE_NAME> RENAME COLUMN APP_PATH_TMP TO APP_PATH;

Table altered.
  • On the Oozie MySQL database - The following example uses the MySQL mysql command-line tool:
    $ mysql -u root -p
Enter password:

mysql> use <OOZIE_DATABASE_NAME>;
Database changed

mysql> ALTER TABLE <TABLE_NAME> MODIFY COLUMN app_path text;
Query OK, X rows affected (0.03 sec)
Records: X Duplicates: 0 Warnings: 0

mysql> exit
Bye
  • On the Oozie MariaDB database - The following example uses the MariaDB mysql command-line tool:
    $ mysql -u root -p
Enter password:

MariaDB [(none)]> use <OOZIE_DATABASE_NAME>;
Database changed

MariaDB [OOZIE_DATABASE_NAME]> ALTER TABLE <TABLE_NAME> MODIFY COLUMN app_path text;
Query OK, X rows affected (2.11 sec) 
Records: X Duplicates: 0 Warnings: 0

MariaDB [OOZIE_DATABASE_NAME]> exit
Bye
  • On the Oozie PostgreSQL database - The following example uses the PostgreSQL psql command-line tool:
    $ psql -U postgres
Password for user postgres: *****

postgres=# \c <OOZIE_DATABASE_NAME>;
You are now connected to database "<OOZIE_DATABASE_NAME>" as user "postgres".

OOZIE_DATABASE_NAME=# ALTER TABLE <TABLE_NAME> ALTER COLUMN app_path type text;
ALTER TABLE

OOZIE_DATABASE_NAME=# \q

Upgrade Oozie SharedLib

  1. Go to the Oozie service.
  2. If the OOZIE service is stopped, start it:

    Select Actions > Start and click Start to confirm.

  3. Select Actions > Install Oozie SharedLib and click Install Oozie SharedLib to confirm.

Upload Tez tar file to HDFS

  1. Go to the TEZ service.

  2. Select Actions > Upload Tez tar file to HDFS and click Upload Tez tar file to HDFS to confirm.

Migrate Hive tables for Cloudera Base on premises upgrade

  1. Go to the HIVE_ON_TEZ service.

  2. Select Actions > Migrate Hive tables for CDP upgrade and click Migrate Hive tables for CDP upgrade to confirm.

Create Ranger Plugin Audit Directory

  1. Go to the Hive-on-Tez service.

  2. Select Actions > Create Ranger Plugin Audit Directory and click Create Ranger Plugin Audit Directory to confirm.

Start Hive on Tez

  1. Go to the Hive-on-Tez service.
  2. Select Actions > Start.

Start Hue

  1. Go to the HUE service.
  2. Select Actions > Start.

Start the Remaining Cluster Services

  1. Use rolling restart or full restart.

  2. Ensure that all services are started or restarted. You can use Cloudera Manager to start the cluster, or you can restart the services individually. The Cloudera Manager Home page indicates which services have stale configurations and require restarting.

  3. To start or restart the cluster:

    1. On the Home > Status page, click the down arrow to the right of the cluster name and select Start or Restart.

    2. Click Start that appears in the next screen to confirm. The Command Details window shows the progress of starting services.

    3. When All services successfully started appears, the task is complete and you can close the Command Details window.

Validate the Hive Metastore Database Schema

  1. Select Actions > Validate Hive Metastore Schema and click Validate Hive Metastore Schema to confirm.
  2. If you have multiple instances of Hive, perform the validation on each metastore database.
  3. Select Actions > Validate Hive Metastore Schema and click Validate Hive Metastore Schema to check that the schema is now valid.

Test the Cluster and Finalize HDFS Metadata

To determine if you can finalize the upgrade, run important workloads and ensure that they are successful. After you have finalized the upgrade, you cannot roll back to a previous version of HDFS without using backups. Verifying that you are ready to finalize the upgrade can take a long time.

When you are ready to finalize the upgrade, do the following:
    1. Go to the HDFS service.
    2. Click the Instances tab.
    3. Click the link for the NameNode instance. If you have enabled high availability for HDFS, click the link labeled NameNode (Active).

      The NameNode instance page displays.

    4. Select Actions > Finalize Metadata Upgrade and click Finalize Metadata Upgrade to confirm.

Clear the Upgrade State Table

After completing all of the previous steps, do the following to complete the upgrade:
  1. Log in to the Cloudera Manager server host.
  2. Stop the Server.
    sudo systemctl stop cloudera-scm-server
  3. Log in to the command-line environment for the Cloudera Manager database. (mysql, sqlplus, or postgres psql).
  4. Run the following command:
    DELETE FROM UPGRADE_STATE;
  5. Start the Server.
    sudo systemctl start cloudera-scm-server