Security-Hardened Spark image migration guide

From Cloudera Data Engineering version 1.24.1, the Redhat-based images for Apache Spark 3.x versions were deprecated and Security-Hardened images are provided for runtime usage. From Cloudera Data Engineering version 1.26.2, the support for the Redhat-based images are removed and only Security-Hardened images are supported for Runtime usage.

Before performing the in-place upgrade or the backup and restore procedure, check Migrating Cloudera Data Engineering Services deployed on deprecated Redhat UBI images to Security-Hardened images.

When manually creating clusters through the Cloudera Data Engineering UI, the API, or the CDP CLI, the default option is the security hardened image.

Steps:

  1. Migrate your workloads to the Security-Hardened images by creating new Virtual Clusters with the Security-Hardened images under the same Cloudera Data Engineering Service.

    For more information, see Migrating Cloudera Data Engineering Services deployed on deprecated Red Hat UBI images to Security-Hardened images.

  2. Perform the in-place upgrade or the backup and restore procedure.
  3. Resolve any incompatibilities relating to your Cloudera Data Engineering applications.

For any packages excluded from the security hardened image, missing for Python or Java modules, you can build a custom image with the required libraries from the security hardened base Spark runtime image. For more information, see Using Custom Spark Runtime Docker Images via API/CLI.