Cloudera Docs

Upgrading to Cloudera Data Lake 7.3.2 with Cloudera Data Engineering

Cloudera Data Engineering 1.25.2 and higher versions support Cloudera Data Lake 7.3.2.

Upgrading to Cloudera Data Lake 7.3.2 is only supported for specific Cloudera Data Lake versions. For more information on the supported Cloudera Data Lake versions, see Upgrading to Cloudera Runtime 7.3.2.

If you plan to upgrade to Cloudera Data Lake 7.3.2 with Cloudera Data Engineering 1.25.2, first, make sure to refactor and migrate all your Spark Jobs to Spark 3.5.4. For more information, see Migrating Apache Spark applications.

Steps

  1. Upgrade to, or create Cloudera Data Engineering Service of version 1.25.0, with Data Lake version 7.2.18 or 7.3.1.

    Data Lake 7.2.18 in Cloudera Data Engineering 1.25.0 supports multiple Spark versions:

    • Spark 2.4.8
    • Spark 3.2.3
    • Spark 3.3.0
    • Spark 3.5.1

    Data Lake 7.3.1 in Cloudera Data Engineering 1.25.0 supports only Spark version 3.5.4.

    For more information, see Compatibility for Cloudera Data Engineering and Runtime components.

  2. In the Cloudera Data Engineering Service, perform the following.
    1. Create new Virtual Clusters (VCs) with Spark 3.5.4.
    2. Refactor your current Spark jobs to Spark 3.5.4.

    For information on the Spark application migration instructions, see Migrating Apache Spark applications.

  3. Test the Spark 3.5.4 Jobs by running them.
  4. After ensuring that everything works as expected, delete the old VCs with lower Spark versions and keep the new VCs.
  5. Upgrade your Cloudera Data Lake version to 7.3.2 in Cloudera Management Console. For more information, see Upgrade Cloudera Runtime to 7.3.2 for the Data Lake clusters.
  6. Perform in-place upgrade on your Cloudera Data Engineering Service to 1.25.2.
  1. Upgrade to, or create Cloudera Data Engineering Service of versions 1.25.2, with Data Lake version 7.2.18 or 7.3.1.

    Data Lake 7.2.18 in Cloudera Data Engineering 1.25.2 supports multiple Spark versions:

    • Spark 2.4.8
    • Spark 3.2.3
    • Spark 3.3.0
    • Spark 3.5.1

    Data Lake 7.3.1 and 7.3.2 in Cloudera Data Engineering 1.25.2 supports only Spark version 3.5.4.

    For more information, see Compatibility for Cloudera Data Engineering and Runtime components.

  2. In the Cloudera Data Engineering Service, perform the following.
    1. Create new Virtual Clusters (VCs) with Spark 3.5.4.
    2. Refactor your current Spark jobs to Spark 3.5.4.

    For information on the Spark application migration instructions, see Migrating Apache Spark applications.

  3. Test the Spark 3.5.4 Jobs by running them.
  4. After ensuring that everything works as expected, delete the old VCs with lower Spark versions and keep the new VCs.
  5. Upgrade your Cloudera Data Lake version to 7.3.2 in Cloudera Management Console. For more information, see Upgrade Cloudera Runtime to 7.3.2 for the Data Lake clusters.
  1. Upgrade to, or create Cloudera Data Engineering Service of versions 1.26.1, with Data Lake version 7.2.18 or 7.3.1.

    Data Lake 7.2.18 in Cloudera Data Engineering 1.26.1 supports multiple Spark versions:

    • Spark 2.4.8
    • Spark 3.2.3
    • Spark 3.3.0
    • Spark 3.5.1

    Data Lake 7.3.1 and 7.3.2 in Cloudera Data Engineering 1.26.1 supports only Spark version 3.5.4.

    For more information, see Compatibility for Cloudera Data Engineering and Runtime components.

  2. In the Cloudera Data Engineering Service, perform the following.
    1. Create new Virtual Clusters (VCs) with Spark 3.5.4.
    2. Refactor your current Spark jobs to Spark 3.5.4.

    For information on the Spark application migration instructions, see Migrating Apache Spark applications.

  3. Test the Spark 3.5.4 Jobs by running them.
  4. After ensuring that everything works as expected, delete the old VCs with lower Spark versions and keep the new VCs.
  5. Upgrade your Cloudera Data Lake version to 7.3.2 in Cloudera Management Console. For more information, see Upgrade Cloudera Runtime to 7.3.2 for the Data Lake clusters.