Security hardened Spark image migration guide

For Apache Spark 3.x runtimes, security-hardened images are recommended. For backward compatibility, you can continue using Red Hat UBI images (also referred to as "Red Hat insecure" images). In Cloudera Data Services on premises 1.5.5, Red Hat UBI images are available for Apache Spark 3.2.x and 3.3.x, while Apache Spark 3.5.x is available exclusively as a security-hardened image. Apache Spark 3.4 is no longer available in version 1.5.5 or later.

To support successful migrations during in-place upgrades and backup-and-restore operations for clusters from previous versions, Redhat images are continued to be used. When manually creating clusters through the Cloudera Data Engineering UI, the API, or the CDE CLI, the default option is the security hardened image.

The security hardened image differs from the Redhat-based images in the following aspects:

• Different Java and Python versions are used. For more information, see Compatibility for Cloudera Data Engineering and Cloudera Runtime components.
• The security hardened image follows a distroless approach, which excludes packages that are not required for running Spark. Several Linux packages which are present in the Redhat-based images are not included in the security hardened image. As a consequence, any Python or Java module that has a dependency on these excluded packages causes issues in the security hardened image. For more information, see Libraries not included in the security hardened image.

For any packages excluded from the security hardened image, missing for Python or Java modules, you can build a custom image with the required libraries from the security hardened base Spark runtime image. For more information, see Using Custom Spark Runtime Docker Images via API/CLI.