CDS 3.1 Powered by Apache Spark Requirements
The following sections describe software requirements for CDS 3.1 Powered by Apache Spark.
CDP Versions
Supported versions of CDP are described below.
| CDS Powered by Apache Spark Version | Supported CDP Versions |
|---|---|
| 3.1.1.3.1.7270.0-253 | CDP Private Cloud Base with Cloudera Runtime 7.1.6 |
A Spark 2 service (included in CDP) can co-exist on the same cluster as Spark 3 (installed as a separate parcel). The two services are configured to not conflict, and both run on the same YARN service. Spark 3 installs and uses its own external shuffle service.
Although Spark 2 and Spark 3 can coexist in the same CDP Private Cloud Base cluster, you cannot use multiple Spark 3 versions simultaneously. All clusters managed by the same Cloudera Manager Server must use exactly the same version of CDS Powered by Apache Spark.
Python Requirement
CDS 3.1 requires Python 3.6+.
JDK Requirement
CDS 3.1 requires JDK 8 or JDK 11. Remove other JDK versions from all cluster and gateway hosts to ensure proper operation.
