Data Lake upgrade

When new versions or builds of Cloudera Runtime/Cloudera Manager are available for the Data Lake service, you can initiate a Data Lake upgrade. An OS upgrade may also be available. Use either the CDP CLI or the Management Console to initiate an upgrade.

There are several potential types of Data Lake upgrades:

Runtime and Cloudera Manager version upgrades, called "major/minor version" upgrades, which are available when a new Runtime and CM version is released.
Service pack upgrades, which do not change the major/minor Runtime or CM release, but upgrade the Data Lake to the latest CM and/or Runtime service packs available in the given version. These upgrades are made available as needed and can deliver targeted bug fixes for Cloudera Runtime, Cloudera Manager, or both.
OS upgrades, which do not change any CM or Runtime builds, but update the underlying cloud image. If an OS upgrade is available along with a major/minor version upgrade, the major/minor upgrade will incorporate the OS upgrade.
Rolling upgrades allow you to upgrade the Data Lake Runtime and OS without stopping the cluster and its services. This means that you can upgrade a Data Lake without stopping the attached Data Hubs and Data Services. Rolling upgrades for the Data Lake are limited to certain Data Lake Runtime versions and shapes. For more information, see Data Lake rolling upgrades.

The type of upgrade that is available for a Data Lake is explicit on the Upgrade tab of the Data Lake details page. Any available upgrades are visible in the Target Cloudera Runtime Version drop-down menu. For example:

Process

The Data Lake upgrade process will:

Check for a newer Cloudera Manager and Runtime version or build, and a new OS image.
Automatically create a backup of the Data Lake (for major/minor and service pack upgrades).
Execute the Data Lake upgrade.
Verify the Data Lake state.

Major/minor version upgrades

Major/minor version upgrades are available as new versions of Cloudera Runtime and Cloudera Manager are released. Version upgrades combine a Runtime and CM upgrade into one operation. For example, this upgrade might involve upgrading from Runtime 7.2.8 to 7.2.9, and Cloudera Manager from 7.4.0 to 7.4.1.

Data Lake version upgrades require you to stop all of the Data Hubs in the environment before performing the Data Lake upgrade. The upgrade process stops all of the Data Lake services, downloads and distributes the new Cloudera Runtime, restarts the services, and deletes the old Cloudera Runtime. This process includes launching entirely new instance(s) from new cloud image(s). Once the Data Lake upgrade is complete, you can then either upgrade your Data Hub clusters to the same version as the Data Lake (if your Data Hub cluster type is supported for upgrade), or delete and recreate the clusters with the new Data Lake version. If your Data Hub is on Runtime version 7.2.16 or later, it is compatible with a Data Lake on a newer Runtime version (7.2.17+). You can independently upgrade your Data Hubs at a later time if you choose to, though it is not required

important

Cloudera Runtime 7.2.17 has new dependencies that were not present in most of the previous Runtime versions. Because of this, you may be unable to upgrade directly to this version and future service packs without first upgrading to a more recent Runtime version.

If you plan to upgrade your existing Data Lakes from a previous release to 7.2.17 or later, you can verify whether or not you will first be required to perform an additional upgrade step:

Select the Data Lake that you want to upgrade and click on the Upgrade tab.
If you see a warning message about missing prerequisites, follow the given steps to perform the additional upgrade before you upgrade to 7.2.17.

If your current Runtime version is 7.2.16 or any 7.2.16 service packs, these additional steps will include first performing an OS upgrade before upgrading to 7.2.17.

If your current Runtime version is 7.2.12, 7.2.14, or 7.2.15, these steps may include upgrading to the most recent service pack of your current Runtime version, as well as performing a separate OS upgrade for your current Runtime version, before you can perform a major/minor version upgrade to 7.2.17.

Service pack upgrades

The service pack upgrade process checks to see if a new Cloudera Manager or Cloudera Runtime (CDP) build is available, and then upgrades the Data Lake to the newest builds. Service pack upgrades do not upgrade to a new version of Cloudera Manager or Cloudera Runtime; they only upgrade to the latest service pack of a particular version. For example, a service pack upgrade cannot take the Data Lake from Runtime version 7.2.11 to version 7.2.12, but instead upgrades Runtime 7.2.11 to a newer 7.2.11 build. These upgrades are made available as needed and can deliver targeted bug fixes for Cloudera Runtime, Cloudera Manager, or both. If desired, specific older service packs can also be chosen using the CDP CLI. The service pack upgrade process follows the same steps as the version upgrade process.

OS upgrades

OS upgrades may be available sporadically as new images are created. OS upgrades are typically released to address security vulnerabilities or other issues on the host system. The OS upgrade process includes launching entirely new instances with the new OS image. An OS upgrade triggers the execution of any pre-service-deployment, post-cluster-manager-start, or post-service-deployment recipes.

Rolling upgrades

Certain Data Lake upgrades can be performed in a rolling fashion, depending on the Data Lake shape, Data Lake OS, and the Runtime version you are upgrading to and from. For more information, see Data Lake rolling upgrades.