Release NotesPDF version

Cloudera Data Engineering on premises 1.5.4 SP1

Review the features, fixes, and known issues in the Cloudera Data Engineering 1.5.4 Service Pack 1 release.

This section lists the new features added in this release for Cloudera Data Engineering on premises.

Support for SAML authentication
Cloudera Data Engineering supports users to configure using SAML as Identity Provider along with LDAP. For more information about how to configure users using SAML authentication, see Configuring users for Identity Providers.
Support for NTP proxy setup
Cloudera Data Engineering requires specific proxy configurations to manage virtual cluster connections efficiently in an air-gapped setup with restricted outbound connections. This setup ensures seamless access to external resources while adhering to network security and management policies. For more information about how to setup NTP proxy, see NTP proxy setup on Cloudera Data Engineering.

This section lists the current known issues and limitations that you might run into while using the Cloudera Data Engineering service.

DEX-15429: Data connectors jobs are failing with "UNKNOWN: Channel Pipeline: [ProtocolNegotiators$ProxyProtocolNegotiationHandler#0, WriteBufferingAndExceptionHandler#0" in the proxy setup
Ozone data connector does not work when proxy is enabled.
Add the IP addresses of all the base cluster nodes to the NO_PROXY list so that request to Ozone does not use proxy.
DEX-15472: Cloudera Data Engineering service upgrade fails during backup and restore when SAML is enabled
Script-based upgrade of Cloudera Data Engineering service doesn't not work when SAML is enabled.
Make sure that the Cloudera Data Engineering service is upgraded using the dex-upgrade-utils solution before enabling the SAML (both IdP and SP initiated) in the Control Plane.
DEX-15735: The dbus-wxm-client pod appears in the dex-base namespace after editing the CPU and Memory quota the first time
After editing the Resource Pool CPU or Memory for a Cloudera Data Engineering Service for the first time, dbus-wxm-client pod gets deployed in the Cloudera Data Engineering Service namespace. The dbus-wxm-client pod keeps failing and cause alerts in the Management Console. However, this will not cause any other issues in Cloudera Data Engineering.
The dbus-wxm-client pod deployment is not needed and can be safely deleted to fix the issue. To delete it, do these steps:
  1. Identify the Cloudera Data Engineering Service namespace:
    1. In the Cloudera console, click the Data Engineering tile. The Cloudera Data Engineering home page displays.
    2. On the left navigation menu, click Administration.
    3. In the Services column, click for the Cloudera Data Engineering service for which you want to identify the namespace.
    4. Note the Cluster ID displayed on the page and identify the Cloudera Data Engineering service namespace. For example, if the Cluster ID is cluster-sales8098, then the Cloudera Data Engineering service namespace is dex-base-sales8098.
    5. Note this namespace.
  2. Using the Cloudera Data Engineering Serivce Namespace, run this Kubernetes command to delete the deployment:
    kubectl delete deployment dex-base-dbus-wxm-client -n [***CDE-SERVICE-NAMESPACE***]
    For example,
    kubectl delete deployment dex-base-dbus-wxm-client -n dex-base-sales8098
DEX-15825: Unable to update the resource quota for Cloudera Data Engineering virtual clusters
Editing the virtual cluster resource quota on Administration > Cluster Details > Configuration > Resource fails for Cloudera Data Engineering. The "Virtual Cluster could not be updated" error message is displayed on the UI if you try to update the virtual cluster CPU, Memory, or the GPU limits.
In the virtual cluster, update the CPU, Memory, and GPU Guaranteed and Maximum values together and click Update.
DEX-15827: In Cloudera Data Engineering on premises, the GPU option is not displayed when creating the virtual cluster
In Cloudera Data Engineering on premises, the GPU option is not displayed for Spark 3.3.2 and Spark 3.2.3 on the Administration > Create a Virtual Cluster UI page.
Create a virtual cluster without GPU. Open the virtual cluster for editing and set the GPU limits.

Review the list of issues that are resolved in the Cloudera Data Engineering service in the Cloudera Data Services on premises 1.5.4 SP1 release.

DEX-14484: Cloudera Data Services on premises- Fix dex-runtime-python-builder to use python 3.8 even for Spark 3.3+
In Spark 3.3.2 virtual cluster, Python 3.8 is now used in Python virtual environment resource instead of old Python 3.6 which is not supported in Spark 3.3 or higher.

We want your opinion

How can we improve this page?

What kind of feedback do you have?