CDS 3 (Experimental) Powered by Apache Spark Requirements

The following sections describe software requirements for CDS 3 (Experimental) Powered by Apache Spark.

CDP Versions

Supported versions of CDP are described below.

CDS Powered by Apache Spark Version Supported CDP Versions
3.0.0.2.99.0.0-23 CDP Data Center 7.0 with Cloudera Runtime 7.0.3

A Spark 2 service (included in CDP) can co-exist on the same cluster as Spark 3 (installed as a separate parcel). The two services are configured to not conflict, and both run on the same YARN service. Spark 3 installs and uses its own external shuffle service.

Although Spark 2 and Spark 3 can coexist in the same CDP Data Center cluster, you cannot use multiple Spark 3 versions simultaneously. All clusters managed by the same Cloudera Manager Server must use exactly the same version of CDS Powered by Apache Spark.

Scala Requirement

CDS 3 requires Scala 2.12.

Python Requirement

CDS 3 requires Python 3.4 or higher.

JDK Requirement

CDS 3 requires JDK 8 or JDK 11. Remove other JDK versions from all cluster and gateway hosts to ensure proper operation.