What's New in Apache Spark

This topic lists new features for Apache Spark in this release of Cloudera Runtime.

Apache Spark version support

Spark included in Cloudera Runtime versions 7.1.1 for CDP Private Cloud Base is based on Apache Spark version 2.4.5 and contains all the feature content of that release.

Dynamic Partition Pruning

Added support for dynamic partition pruning. It is disabled by default.

For more information, see SPARK-11150.