What's new in 1.5.5 SP1

Cloudera Data Engineering on premises 1.5.5 SP1 delivers a set of new features for Cloudera Data Engineering.

Migrating Cloudera Data Engineering Virtual Clusters to Spark 3.5 after upgrading Cloudera Data Lake to 7.3.1

In Cloudera Data Engineering, Cloudera Data Lake 7.3.1 only supports Apache Spark version 3.5. If you upgraded your Cloudera Data Services on premises to 1.5.5 CHF1 or a higher version and your Cloudera Data Lake version is lower than 7.3.1, then the Cloudera Data Engineering Virtual Clusters are running on Spark version lower than 3.5. After upgrading Cloudera Data Lake to 7.3.1, Cloudera Data Engineering Virtual Clusters running on Spark versions lower than 3.5 will not work. For seamless experience, migrate your Cloudera Data Engineering VCs to Spark 3.5. For instructions on migrating Cloudera Data Engineering Virtual Clusters to Spark 3.5, see Migrating Virtual Clusters to Spark 3.5 after upgrading Cloudera Data Lake to 7.3.1.

Support for adding certificates while creating a Cloudera Data Engineering repository

In Cloudera Data Engineering 1.5.5 SP1 and higher versions, Cloudera Data Engineering service supports adding TLS certificates while creating a repository through Cloudera Data Engineering UI or CDE CLI. For instructions, see

• Creating a repository in Cloudera Data Engineering UI [Technical Preview]
• Creating a repository in Cloudera Data Engineering using the CLI

Hadoop authentication using CDE CLI

In Cloudera Data Engineering 1.5.5 SP1 and higher versions, Cloudera Data Engineering service supports configuring or updating Hadoop Authentication using CDE CLI also. For more information, see Hadoop Authentication.

Password based keytab generation for Hadoop Authentication

In Cloudera Data Engineering 1.5.5 SP1 and higher versions, Cloudera Data Engineering service supports configuring Hadoop Authentication using principal and password also. For instructions on configuring Hadoop Authentication using password, see Configuring Hadoop Authentication.

Support for artifact sharing through Cloudera Data Engineering UI

In Cloudera Data Engineering 1.5.5 SP1 and higher versions, you can share artifacts such as jobs, repositories, resources, and credentials with other users or groups using Cloudera Data Engineering UI as well. Using Cloudera Data Engineering UI, you can perform the following actions:

• Share jobs and sessions with other users or groups while creating them. For more information, see Sharing an artifact with a user or group.
• Share jobs, repositories, and resources while updating them. For more information, see Sharing an artifact with a user or group.
• Stop sharing jobs, repositories, and resources, while updating them. For more information, see Stop sharing an artifact with a user or group.

The Artifact Sharing access of a Cloudera Data Engineering job also applies to the job details in the Spark history server UI. For more information, see Using Spark history server to troubleshoot Spark jobs.

Creating multiple connections to various services using Airflow UI

You can create a connection from your Cloudera Data Engineering Virtual Clusters for Sqoop operator or to Cloudera Data Warehouse or Cloudera Base on premises Hive or Impala using SQL operators. For instructions, see

• Creating a connection to Cloudera Data Warehouse Virtual Warehouse
• Creating a connection to Cloudera Base on premises
• Creating a connection for Sqoop Operator

Increased default Airflow worker pod resource request limits

The default values for Airflow worker pod resource request limits are updated the following ways:

• CPU requests increased from 100 m to 1.
• CPU limits increased from 1 to no limit.
• Memory requests increased from 200 Mi to 2 Gi.
• No change in Memory limit. It is 2 Gi.

For more information, see Automating data pipelines using Apache Airflow in Cloudera Data Engineering.

Apache Airflow version upgrade to 2.11.0

The Airflow version that Cloudera Data Engineering uses is upgraded to Airflow 2.11.0. For more information, see:

• Apache Airflow 2.11.0 Release Notes
• Compatibility for Cloudera Data Engineering and Cloudera Runtime components

Kubernetes version upgrade to 1.31

The Kubernetes version that Cloudera Data Engineering uses is upgraded to Kubernetes 1.31. For more information, see Compatibility for Cloudera Data Engineering and Cloudera Runtime components.