Overview

Apache Spark is a general framework for distributed computing that offers high performance for both batch and interactive processing. It exposes APIs for Java, Python, and Scala. This document describes CDS (Cloudera Distribution of Spark), which enables you to install and evaluate the Apache Spark 3.x without upgrading your CDP Private Cloud Base cluster.

CDS (Cloudera Distribution of Spark) is an add-on service for CDP Private Cloud Base, distributed as a parcel and the Cloudera Service Descriptor file is available in Cloudera Manager for CDP.

On CDP Private Cloud Base, a Spark 3 service can coexist with the existing Spark 2 service. The configurations of the two services do not conflict and both services use the same YARN service. The port of the Spark History Server is 18088 for Spark 2 and 18089 for Spark 3.

important

In July 2025, Cloudera released CDS 3.5 (Cloudera Distribution of Apache Spark) for Cloudera on-premises 7.1.9 in July 2025. This release will provide long-term support for Apache Spark 3.5, the latest Long-Term Support (LTS) version adopted by both Cloudera and the broader Apache community. Spark 3.5 is expected to deliver a performance boost over 20% compared to Spark 3.3, along with enhanced security, support for newer Python versions, and bug fixes.

CDS 3.5 on 7.1.9 enables a smooth transition path to the Cloudera 7.3.x releases, which ship with Spark 3.5 as the default.

As part of this transition, CDS 3.3 will be deprecated with support scheduled to end in August 2026. Customers currently using CDS 3.3 on existing 7.1.9 clusters are advised to install the CDS 3.5 package and migrate their workloads to Spark 3.5 to benefit from the latest Spark capabilities and enjoy continued support from Cloudera.

For more information on CDS 3.5, refer to CDS 3.5 Powered by Apache Spark Overview.

For more information on CDS 3.3, refer to CDS 3.3 Powered by Apache Spark Overview.