Behavioral Changes in Spark

Cloudera Runtime 7.3.2.20000 SP2

There are no behavioral changes in this release.

Cloudera Runtime 7.3.2.10000 SP1

CDPD-98799: Rebase on Apache Spark 3.5.8

Previous behavior:

Bundled Spark in Cloudera Runtime 7.3.2.0 was based on Apache Spark 3.5.4.

New behavior:

From 7.3.2.10000 SP1, bundled Spark is based on Apache Spark 3.5.8. Even maintenance rebases can affect application code or defaults; review Upgrading Apache Spark 3.5.4 to Spark 3.5.8, Migrating Spark applications, and the Apache Spark 3.5.8 documentation before production rollout.

When you enable Encrypt all ports in Cloudera Manager, Cloudera Manager applies the Spark property changes in this section automatically. For an overview, see Encryption in transit for Spark and Livy in Cloudera Runtime 7.3.2.10000 SP1.

OPSAPS-76333: Spark authentication and network encryption when Encrypt all ports is enabled

Previous behavior:

spark.authenticate and spark.network.crypto.enabled were not enabled automatically when you turned on cluster-wide port encryption.

New behavior:

When Encrypt all ports is active, Cloudera Manager sets spark.authenticate and spark.network.crypto.enabled to secure Spark RPC with AES encryption. See Configuring Spark for Wire Encryption.

OPSAPS-75940: Spark IO encryption key size and algorithm defaults in Cloudera Manager

Previous behavior:

Cloudera Manager exposed spark.io.encryption.keySizeBits with a default of 128 and did not surface spark.io.encryption.keygen.algorithm on the Spark configuration page.

New behavior:

Cloudera Manager sets the default spark.io.encryption.keySizeBits to 256 and the default spark.io.encryption.keygen.algorithm to HmacSHA256. These defaults apply when you enable optional on-disk IO encryption.

CDPD-94374, OPSAPS-75458: HTTPS-only Spark History Server and Spark UI when Encrypt all ports is enabled

Previous behavior:

When TLS was enabled, Jetty could still bind an HTTP port for the History Server or Spark UI unless you configured HTTPS-only listeners manually.

New behavior:

When Encrypt all ports is active and TLS is enabled, Cloudera Manager sets spark.ssl.historyServer.disableHttpPort and spark.ssl.ui.disableHttpPort to true. Jetty starts with HTTPS only. If TLS is not enabled, these properties are ignored.

CDPD-87464, OPSAPS-76633: spark.cloudera.auto.disable.insecure.spark.ui.enabled

Previous behavior:

The Spark UI could start without SSL even when insecure UI access was not intended.

New behavior:

When this property is enabled, Spark disables the UI if SSL is not active. The parameter is off by default and is enabled automatically when Encrypt all ports is active.

CDPD-87464, OPSAPS-76633: YARN auto-certificate for Spark UI in cluster mode

Previous behavior:

Spark did not use YARN ResourceManager ApplicationMaster certificates for the Spark UI in cluster mode unless you configured certificate policy manually.

New behavior:

When spark.cloudera.ui.yarn.autoCertificate.clusterMode.enabled is enabled, Spark can use a certificate issued to the ApplicationMaster for the YARN RM Proxy. This requires yarn.resourcemanager.application-https.policy set to LENIENT or STRICT in yarn-site.xml. When Encrypt all ports is active, Cloudera Manager sets the YARN policy to LENIENT automatically.

This feature applies only in cluster deploy mode. It does not apply in client mode because the driver runs on the submission host and the UI starts before YARN provides a certificate.

Summary: Default JVM trust store password for BCFKS cacerts

Previous behavior:

Spark did not set javax.net.ssl.trustStorePassword when the JVM default trust store was BCFKS, which could cause Spark History Server or jobs to fail on FIPS 140-3 clusters.

New behavior:

When spark.ssl.defaultTrustStorePassword is set, Spark supplements driver and executor JVM options and applies the password when Spark History Server creates a SecurityManager. See Spark and Livy on FIPS 140-3 clusters.

Cloudera Runtime 7.3.2.100 CHF 1

There are no behavioral changes in this release.

Cloudera Runtime 7.3.2

Cloudera Runtime 7.3.2 introduces functional adjustments, behavioral updates for Spark, and includes all service packs and cumulative hotfixes from 7.3.1.100 through 7.3.1.706. For a comprehensive record of all functional adjustments in Cloudera Runtime 7.3.1.x, see Spark Behavioral Changes.

Summary: Enhanced the security of Spark by implementing new default configuration settings

Previous behavior:

1. spark.ui.enabled=true: possible initiation of an HTTP service that can be accessed from external hosts.
2. spark.io.encryption.keySizeBits=128: the default Spark keySizeBits

New behavior:

1. spark.ui.enabled=false: preventing initiation of an HTTP service that can be accessed from external hosts.
2. spark.io.encryption.keySizeBits=256: the default Spark keySizeBits has been increased from 128 to 256