Cloudera Docs

Performing a Data Hub OS upgrade

Perform a Data Hub OS upgrade to update the OS and VM packages to those available in the latest pre-warmed image.

Required role: DatahubAdmin or Owner over the Data Hub

It your cluster uses Streams Replication Manager (SRM), it is likely that you are storing security files, like truststores, in /opt/cloudera/security/ on your cluster hosts. These files are lost during the upgrade. As a result, you must back them up before the upgrade and restore them following the upgrade.

Before performing any OS upgrade, make sure there is no data belonging to NiFi or NiFi Registry on the root disk of the VM (this is the case for any version before CDP 7.2.10).

If you have NiFi or NiFi Registry data on the root disk, run the commands below to move the data to the right location prior to performing the upgrade. Before executing these scripts, stop the NiFi and NiFi Registry services.

On the NiFi nodes:

When upgrading from CDP 7.2.9 or lower versions

hadoopDirectory=/hadoopfs/$(ls /hadoopfs/ | sort | tail -n 1)
mkdir $hadoopDirectory/working-dir
cp -R /var/lib/nifi/* $hadoopDirectory/working-dir
echo "nifi.working.directory should be set to $hadoopDirectory/working-dir"latest_nifi_conf_directory=$(find /run/cloudera-scm-agent/process -name nifi\.properties | sort | tail -n 1)
latest_nifi_conf_directory=${latest_nifi_conf_directory%nifi.properties}
echo "Latest nifi conf directory used to copy files before migration: $latest_nifi_conf_directory"mkdir -p $hadoopDirectory/working-dir/config_backup
chmod 755 $hadoopDirectory/working-dir/config_backup
cp $latest_nifi_conf_directory/nifi.properties $hadoopDirectory/working-dir/config_backup
cp $latest_nifi_conf_directory/bootstrap.conf $hadoopDirectory/working-dir/config_backup
chown nifi:nifi -R $hadoopDirectory/working-dir

After the bash script executes, update the nifi.working.directory configuration value with what the script returns.

To set nifi.working.directory, perform the following steps:

  1. Open the Cloudera Manager UI.
  2. Go to the NiFi service.
  3. Select the Configuration tab.
  4. Search for nifi.working.directory.
  5. Set the new value and click Save.

On the management node (where NiFi Registry is):

hadoopDirectory=/hadoopfs/$(ls /hadoopfs/ | sort | tail -n 1)
mkdir $hadoopDirectory/working-dir
cp -R /var/lib/nifiregistry/* $hadoopDirectory/working-dir
chown nifiregistry:nifiregistry -R $hadoopDirectory/working-dir
echo "nifi.registry.working.directory should be set to
$hadoopDirectory/working-dir"

After the bash script executes, update the nifi.registry.working.directory configuration value with what the script returns.

To set nifi.registry.working.directory, perform the following steps:

  1. Open the Cloudera Manager UI.
  2. Go to the NiFi service.
  3. Select the Configuration tab.
  4. Search for nifi.registry.working.directory.
  5. Set the new value and click Save.

After the new configuration values are set on the NiFi and the management nodes, start the services.

  1. In the left-hand menu of CDP, click Data Hubs and then select the Data Hub to upgrade.
  2. Scroll to the bottom of the Data Hub details page and select the Upgrade tab.
  3. From the drop-down menu, select the Target Runtime Version. An OS upgrade is indicated by the text “OS Upgrade only” next to the target version.
  4. Click Upgrade.
  5. Monitor the upgrade progress using the Data Hub Event History tab.

If you upgraded from CDP 7.2.9 or a lower version, perform the following tasks on the NiFi UI as post-upgrade steps when NiFi is available and running.

  1. Open the NiFi UI.
  2. Select Controller Settings from the Global Menu available in the top-right corner of the NiFi UI.
  3. Go to the Management Controller Services tab of the NiFi Settings dialog.
  4. Edit the Default Reporting Task SSL Context Service.
    1. Disable the running controller service by using the button.
    2. Modify the Keystore Filename value to point to the new nifi working directory. Use /hadoopfs/fs4/working-dir/cm-auto-host_keystore.jks instead of /var/lib/nifi/cm-auto-host_keystore.jks.
    3. Click APPLY.
  5. Enable the Default Reporting Task SSL Context Service using the button.
  6. Go to the Reporting Tasks tab.
  7. Edit the Default Atlas Reporting Task by clicking the button.
    1. Modify the Atlas Configuration Directory to point to the new nifi working directory. Use /hadoopfs/fs4/working-dir/ instead of /var/lib/nifi/.
    2. Modify the Kerberos Keytab to point to the new nifi working directory. Use /hadoopfs/fs4/working-dir/nifi.keytab instead of /var/lib/nifi/nifi.keytab.
    3. Click APPLY.

If your cluster uses SRM, restore the security files you backed up before the upgrade.