Cloudera Docs

Spark 3 support in Oozie

This section describes how to re-enable Spark actions, migrate workflows from Spark actions to Spark 3 actions, and align Cloudera Runtime and CDS versions when you run Spark 3 through Oozie.

Due to the discontinuation and deprecation of Spark 2, Oozie's Spark actions are deprecated, which are based on Spark 2. Consequently, Oozie's Spark actions are disabled by default, and if you attempt to execute a Spark action, an error appears. Oozie supports the new Spark 3 based Spark 3 actions.

Compatibility between Cloudera Runtime and CDS

Oozie Spark 3 actions load libraries from both the CDP cluster (Oozie and Hadoop) and the CDS (Spark 3) parcel. If the Cloudera Runtime and CDS versions are not aligned, Spark 3 jobs that you submit through Oozie can fail at runtime with classpath errors. The following sections describe known mismatches on Cloudera Runtime 7.1.9 SP2.

Commons Lang (7.1.9 SP2)

When you run CDP Private Cloud Base with Cloudera Runtime 7.1.9 SP2 (including any CHF) and a CDS parcel that was released before CDS 3.3 for 7.1.9 SP2 or CDS 3.5 for 7.1.9 SP2, Oozie Spark 3 jobs can fail. Cloudera Runtime 7.1.9 SP2 updates Apache Commons Lang on the cluster, and the CDS parcel must include the matching libraries.

Typical errors include NoSuchMethodError on org.apache.commons.lang.StringUtils.join, or ClassNotFoundException for org.apache.commons.lang.StringEscapeUtils.

To resolve this issue, activate CDS 3.3 for 7.1.9 SP2 or CDS 3.5 for 7.1.9 SP2. Do not use a CDS parcel from the 7.1.9 SP1 line or another service pack when the cluster runs 7.1.9 SP2.

Commons Configuration (7.1.9 SP2 CHF2)

Cloudera Runtime 7.1.9 SP2 CHF2 upgrades Commons Configuration in the CDP stack. Oozie from 7.1.9 SP2 CHF2 is compatible only with a CDS parcel that includes the matching Commons Configuration2 changes for 7.1.9 SP2.

When versions are mismatched, Spark 3 jobs launched from Oozie can fail during Spark Atlas Connector initialization with an error similar to the following:

java.lang.NoSuchMethodError: org.apache.atlas.ApplicationProperties.get()Lorg/apache/commons/configuration/Configuration;

On Cloudera Runtime 7.1.9 SP2 CHF2, use CDS 3.3 CHF2 for 7.1.9 SP2 or CDS 3.5 CHF2 for 7.1.9 SP2. Do not use an earlier CDS cumulative hotfix from the 7.1.9 SP2 parcel line.