Cloudera Docs

CDS 3.2.3 Powered by Apache Spark Overview

Apache Spark is a general framework for distributed computing that offers high performance for both batch and interactive processing. It exposes APIs for Java, Python, and Scala.

For detailed API information, see the Apache Spark project site.

CDS 3.2.3 Powered by Apache Spark is an add-on service for CDP Private Cloud Base, distributed as a parcel and custom service descriptor.

This document describes CDS 3.2.3 Powered by Apache Spark. CDS (Cloudera Distribution of Spark) enables you to install and evaluate the features of Apache Spark 3 without upgrading your CDP Private Cloud Base cluster.

On CDP Private Cloud Base, a Spark 3 service can coexist with the existing Spark 2 service. The configurations of the two services do not conflict and both services use the same YARN service. The port of the Spark History Server is 18088 for Spark 2 and 18089 for Spark 3.

CDS 3 for GPUs

CDS 3.2.3 with GPU Support is an add-on service that enables you to take advantage of the RAPIDS Accelerator for Apache Spark to accelerate Apache Spark 3 performance on existing CDP Private Cloud Base clusters.

Unsupported connectors

This release does not support the following connectors:

  • Phoenix
  • SparkR
  • Hive Warehouse
  • Oozie
  • Zeppelin

Unsupported Features

This release does not support the following feature:

  • Hudi
  • Push-based shuffle

Limitations of Spark in CDP

Limitations of Spark (in comparison to Apache Spark 3.2) in CDP are described below:

  • spark.sql.orc.compression.codec config doesn't accept zsdt value.
  • spark.sql.avro.compression.codec config doesn't accept zstandard value.
  • Specifying avroSchemaUrl is not supported in datasource options.
  • Spark3 parcel is not available on the IBM PowerPC platform.