Cloudera Docs

Behavioral Changes in Spark

Behavioral changes denote a marked change in behavior from the previously released version to this version of Spark.

Cloudera Runtime 7.3.1.700 SP3 CHF 2

There are no behavioral changes in this release.

Cloudera Runtime 7.3.1.600 SP3 CHF 1

Summary:
Added the spark_shuffle_aes_enabled configuration property
Previous behavior:
None.
New behavior:
The spark_shuffle_aes_enabled new configuration property is now available to enable AES for the Spark shuffle service.
Summary:
Added the spark3.network.crypto.enabled configuration property
Previous behavior:
None.
New behavior:
The spark3.network.crypto.enabled new configuration property is now available to enable AES-based encryption.
Summary:
Changes in the generation of the spark.yarn.historyServer.address value
Previous behavior:
Configuration files displayed Spark History Server URL as HTTP.
New behavior:
The Spark CSD generates the configuration with the HTTPS address when SSL/TLS is enabled.

Cloudera Runtime 7.3.1.500 SP3

Summary:
Added super-interface java.nio.file.PathMatcher to the IOFileFilter API.
Previous behavior:
None.
New behavior:
Unless updated to use the new super-interface, recompilation of a client program may be terminated with the following message: A client class C is not abstract and does not override abstract method in java.nio.file.PathMatcher.
Summary:
Added java.io.IOException exception thrown to the CountingInputStream class in the CountingInputStream.afterRead API.
Previous behavior:
None.
New behavior:
Unless updated to handle the new exception, recompilation of a client program may be terminated with the following message: Unreported exception java.io.IOException must be caught or declared to be thrown.
Summary:
Added java.io.FileNotFoundException exception thrown to the FileUtils.isFileOlder class in the FileUtils API.
Previous behavior:
None.
New behavior:
Unless updated to handle the new exception, recompilation of a client program may be terminated with the following message: Unreported exception java.io.FileNotFoundException must be caught or declared to be thrown.
Summary:
Return value type has been changed in the StreamIterator class in the StreamIterator.iterator API.
Previous behavior:
Return value was java.util.Iterator<T>.
New behavior:
New return value is org.apache.commons.io.StreamIterator<T>
Summary:
Removed the java.io.EOFException exception thrown in the SwappedDataInputStream class in the SwappedDataInputStream.skipBytes API.
Previous behavior:
Application code could handle the java.io.EOFException exception thrown.
New behavior:
Unless removed from the application code, recompilation of a client program may be terminated with the following message: Cannot override <span class='iname_b'>skipBytes&#160;<span class='sym_pd'><span>(&#160;int lt;/span>&#160;)</span></span> in <b>org.apache.commons.io.input.SwappedDataInputStream</b>; overridden method does not throw java.io.EOFException.
Summary:
Return value type has been changed in the StreamIterator class in the StreamIterator.iterator API.
Previous behavior:
Return value was java.util.Iterator<T>
New behavior:
New return value is org.apache.commons.io.StreamIterator<T>.
The previous method has been removed because the return type is part of the method signature. A client program may be interrupted by a NoSuchMethodError exception.

Cloudera Runtime 7.3.1.400 SP2

There are no behavioral changes in this release.

Cloudera Runtime 7.3.1.300 SP1 CHF 1

There are no behavioral changes in this release.

Cloudera Runtime 7.3.1.200 SP1

Summary:
Rebase Spark3 to Apache Spark 3.5.4 in Cloudera Runtime.
Previous behavior:
Spark 3.4.1 was the default version in Cloudera Runtime.
New behavior:

Spark 3.5.4 is the default Spark version in Cloudera Runtime.

Cloudera Runtime 7.3.1.100 CHF 1

There are no behavioral changes in this release.

Cloudera Runtime 7.3.1

Summary:
Spark 2 has been removed from Cloudera Runtime.
Previous behavior:

Spark 2 was the default version in Cloudera Runtime, Spark 3 was available as an add-on parcel.

New behavior:

Spark 3 is the default Spark version in Cloudera Runtime. Spark 2 has been removed and no longer available in 7.3.1.0.