Cloudera Docs

Upgrade Spark3 within Oozie

In Cloudera Private Cloud Base, Spark 3 is not included in the CDH parcel but is available as a separate parcel. This allows independent upgrades of Spark 3.

When upgrading the Spark 3 parcel, you need to update the Spark 3 distribution in Oozie's Spark 3 ShareLib since Oozie's Spark 3 ShareLib is created during the Oozie build process.

To simplify this process, you can use the Update the Spark3 ShareLib action in Cloudera Manager:


This command performs an update of the Spark 3 ShareLib based on the Spark 3 parcel, while retaining all other ShareLibs in their current state. This ensures that any custom JARs placed in other ShareLibs are not deleted. However, if you previously placed custom JARs in Oozie's Spark 3 ShareLib, they are deleted during this operation. In such cases, you need to manually upload those JARs to HDFS again.
To maintain compatibility, the existing Install Oozie ShareLib action also updates the Spark 3 ShareLib. However, you can disable this feature using the following setting:


By default, the Update the Spark3 ShareLib action selects the currently activated Spark 3 parcel for uploading JARs. However, you can modify this behavior using the following setting:


The setting should specify the name of the folder for the Spark 3 parcel to use, located at the root directory of parcels, typically /opt/cloudera/parcels. By default, it is set as SPARK3, which is a symbolic link to the currently activated Spark 3 parcel. You can also specify the folder name of another downloaded and distributed Spark 3 parcel that is not currently activated. For instance, SPARK3-3.3.0.3.4.7190.0-174-1.p0.42371348. Cloudera recommends to change the default setting only if you encounter any problem with the currently activated Spark 3 parcel in Oozie. While your issue is being investigated, you can temporarily use a different version of Spark 3 with Oozie by modifying this setting.

Please be aware that if you change the default value of this setting, Cloudera Manager displays a warning to remind you about this: