What's New in Spark

Cloudera Runtime 7.3.2.20000 SP2

There are no new features in this release.

Cloudera Runtime 7.3.2.10000 SP1

Rebase on Apache Spark 3.5.8

Spark shipped with Cloudera Runtime 7.3.2.10000 SP1 is based on Apache Spark 3.5.8 (previously 3.5.4 in Cloudera Runtime 7.3.2). This maintenance rebase includes bug fixes and security updates from upstream Apache Spark. For more information, see the Apache Spark documentation.

See Upgrading Apache Spark 3.5.4 to Spark 3.5.8 and Migrating Spark applications.

FIPS 140-3 support for Spark 3

Spark 3 supports FIPS 140-3 on FIPS-enabled clusters in this service pack. Spark cryptographic modules and Cloudera SASL are updated for FIPS 140-3 validation.

For configuration details, see Spark and Livy on FIPS 140-3 clusters.

Encryption in transit for Spark

Spark 3 supports stronger encryption in transit when you enable Encrypt all ports in Cloudera Manager, including Spark network encryption, HTTPS-only History Server and UI listeners, and updated IO encryption defaults. Spark History Server and UI TLS can use TLS 1.3 when your Cloudera Manager TLS policy allows it.

For configuration and behavioral changes, see Encryption in transit for Spark and Livy in Cloudera Runtime 7.3.2.10000 SP1, Configuring TLS/SSL encryption manually for Spark, and Behavioral Changes in Spark.

Cloudera Runtime 7.3.2.100 CHF 1

There are no new features introduced in this release.

Cloudera Runtime 7.3.2.0

Rebase Spark3 to Apache Spark 3.5.4 in Cloudera Runtime.

Spark 3.5.4 is the default Spark version in Cloudera Runtime. Refer to the Apache Spark documentation for more information on changes.

Spark 3 is the default in Cloudera Runtime

Spark 3 is the default Spark version in Cloudera Runtime. Spark 2 has been removed and no longer available in 7.3.2.0