What's New in Apache Spark

This topic lists new features for Apache Spark in this release of Cloudera Runtime.

Apache Spark version support

Spark included in Cloudera Runtime versions 7.1.1 and later for CDP Private Cloud Base is based on Apache Spark version 2.4.5 and contains all the feature content of that release.

Data engineering cluster

You can create a data engineering cluster in Amazon AWS from within CDP by selecting the Data Engineering cluster template. A data engineering includes Spark, Livy, Hive, Zeppelin, and Oozie, along with supporting services (HDFS, YARN, and Zookeeper).

See Creating a Cluster on AWS.