CDS 3.1 Powered by Apache Spark Requirements

The following sections describe software requirements for CDS 3.1 Powered by Apache Spark.

CDP Versions

Supported versions of CDP are described below.

CDS Powered by Apache Spark Version Supported CDP Versions
3.1.1.3.1.7270.0-253 CDP Private Cloud Base with Cloudera Runtime 7.1.6 and higher

A Spark 2 service (included in CDP) can co-exist on the same cluster as Spark 3 (installed as a separate parcel). The two services are configured to not conflict, and both run on the same YARN service. Spark 3 installs and uses its own external shuffle service.

Although Spark 2 and Spark 3 can coexist in the same CDP Private Cloud Base cluster, you cannot use multiple Spark 3 versions simultaneously. All clusters managed by the same Cloudera Manager Server must use exactly the same version of CDS Powered by Apache Spark.

Scala Requirement

CDS 3.1 requires Scala 2.12.

Python Requirement

CDS 3.1 requires Python 3.6+.

JDK Requirement

CDS 3.1 requires JDK 8 or JDK 11. Remove other JDK versions from all cluster and gateway hosts to ensure proper operation.