CDS 3.1 for GPUs requirements

Hardware requirements

CDS 3.1 for GPUs requires cluster hosts with NVIDIA Pascal™or better GPUs, with a compute capability rating of 6.0 or higher.

For more information, see Getting Started at the RAPIDS website.

Cloudera and NVIDIA recommend using NVIDIA-certified systems. For more information, see NVIDIA-Certified Systems in the NVIDIA GPU Cloud documentation.

Software requirements

Each cluster host with a GPU must have the following software installed:

Java

JDK 8 or JDK 11. Cloudera recommends using JDK 8, as most testing has been done with JDK 8. Remove other JDK versions from all cluster and gateway hosts to ensure proper operation.

Scala

Scala 2.12

Python

Python 3.6 and higher

GPU drivers and CUDA toolkit

GPU driver v450.80.02 or higher

CUDA version 11.0 or higher

Download and install the CUDA Toolkit for your operating system. The toolkit installer also provides the option to install the GPU driver.

(Optional) UCX

UCX 1.10.1 required

Clusters with Infiniband or RoCE networking require Unified Communication X (UCX) to enable the RAPIDS Shuffle Manager. For instructions on installing UCX native libraries, see (Optional) Installing UCX native libraries.

CDP versions

Supported versions of CDP are described below.

The Spark 2 service (included in CDP) can co-exist on the same cluster as a Spark 3 service (including CDS 3.1 for GPUs, installed as a separate parcel). The two services are configured to not conflict, and both run on the same YARN service. CDS 3.1 for GPUs installs and uses its own external shuffle service.

Although Spark 2 and Spark 3 can coexist in the same CDP Private Cloud Base cluster, you cannot use multiple Spark 3 versions simultaneously. All clusters managed by the same Cloudera Manager Server must use exactly the same version of CDS, whether that is CDS Powered by Apache Spark or CDS 3.1 for GPUs.