Homepage

Cloudera Public Cloud: November 2024 Release Summary

The Release Summary of Cloudera Public Cloud summarizes major features introduced in Cloudera Management Console, Cloudera Data Hub, and data services.

Cloudera AI

The 2.0.46-b238 release of the Cloudera AI on Public Cloud introduces the following changes.

New Features / Improvements

  • Model Hub Enhancement: The model size is now shown in the user-friendly format both in the Model Hub UI and AI Registry UI.
  • Cloudera AI Inference service Enhancement: New AI Inference Services menu item is added to the left-navigation pane of the Cloudera AI UI to manage the lifecycle of Cloudera AI Inference service using UI. For more information, see Managing Cloudera AI Inference service using the UI.
  • Added Spark 3.5 ML Runtime Addon
  • Product and features names:
    • Cloudera Machine Learning (CML) is renamed to Cloudera AI.
    • Cloudera Machine Learning Model Registry is renamed to Cloudera AI Registry.
    • Cloudera Machine Learning Workspaces is renamed to Cloudera AI Workbenches.
    • Cloudera Applied Machine Learning Prototypes and Accelerators for ML Projects is renamed to
    • Cloudera Accelerators for Machine Learning Projects.

Fixed Issues

  • CVE fixes - This release includes numerous security fixes for critical and high Common Vulnerability and Exposures (CVE).
  • Previously, the public and private settings did not carry forward after the AI Registry upgrade. This issue is now resolved. (DSE-36799)
  • Enhanced the error message that was displayed when importing a model from Model Hub to Registered Models. (DSE-39897)
  • Generic (vLLM) NIM profile deployment was returning an empty GPU list in the UI. This issue is now resolved. (DSE-39913)
  • Previously, public cloud CDP CLI was not showing the instance type’s GPU count. This issue is now resolved. (DSE-39539)
  • Cloudera AI v2 API deployed application did not inherit user-level environment variables and site-level environment variables. This issue has been solved, and now an application created using APIv2 does not only inherit project-level environment variables but also user-level environment variables and site-level environment variables. (DSE-37611)
  • Previously, scheduled jobs skipped job runs and did not specify the error. Now, the skipped jobs runs have improved exit code to distinguish them from failed jobs. (DSE-39976)
  • Previously, the Next buttons on the Site Administration page did not work. This issue is now resolved. (DSE-34133).
  • Previously, restarting the application using the Cloudera AI v2 API did not inherit account application-level environment variables. This issue is now resolved. (DSE-39894)
  • Users can now view the existing applications in the Cloudera AI UI even if the creation of a new application is disabled. (DSE-39980)
  • Previously, Python logging did not work with PBJ Runtimes. This issue is now resolved. (DSE-39929)
  • Previously, reloading the session page would result in an incorrect state where the PBJ session’s editor cell could appear green even if it is in a processing state (executing some commands). With this fix, an accurate representation of the processing state is displayed even after a refresh. (DSE-40049)

Cloudera Data Engineering

The 1.23.0 release of the Cloudera Data Engineering service on Public Cloud introduces the following changes.

External IDE connectivity through Spark Connect-based sessions (Technical Preview)
Cloudera Data Engineering supports Spark Connect sessions, which are Cloudera Data Engineering sessions that expose the Spark Connect interface. Spark Connect sessions allow you to run Spark commands from any remote Python environment. For more information, see External IDE connectivity through Spark Connect-based sessions.

In-place upgrade enhancements
In-place upgrade enhancements have been documented to improve handling Airflow version 2.9 and Python version 3.11 changes during the upgrade. For more information, see In-place upgrade with Airflow Operators and Libraries.

Backup-restore-based upgrade enhancements
Backup-restore-based upgrade enhancements have been implemented.

Airflow version upgrade to 2.9
The Airflow version that Cloudera Data Engineering uses is upgraded to 2.9. For more information, see:

Python 3.11 support for Apache Airflow
The Python version that Cloudera Data Engineering uses is upgraded to 3.11. For more information, see Compatibility for Cloudera Data Engineering and Runtime components.

Iceberg version upgrade to 1.5
The Iceberg version that Cloudera Data Engineering uses is upgraded to 1.5.2. For more information, see: Compatibility for Cloudera Data Engineering and Runtime components.

Kubernetes version upgrade to 1.30
The Kubernetes version that Cloudera Data Engineering uses is upgraded to 1.30. For more information, see Compatibility for Cloudera Data Engineering and Runtime components.

Deprecation of CDW Operator
Starting from Cloudera Data Engineering service version 1.23.0, CDW Operator is deprecated. Cloudera recommends you to use SQL Operator in place of CDW Operator. For more information, see Creating a connection to Cloudera Data Warehouse for CDW Operator.

Active node count
The active node count has been removed from the following Cloudera Data Engineering UI pages:

  • Administrator Overview
  • Administrator Overview > Service Details

For real-time node count information, check your Cloud Service Provider’s website. For more information, see Checking the node count on your Cloud Service provider’s website.

Fixed Issues
DEX-9617: CDE Jobs Page-Airflow UI: Audit Log tab does not allow further navigation The fix allows you to view the Airflow DAG run audit logs.

Cloudera DataFlow

The 2.9.0-h3-b1 release of Cloudera DataFlow fixes a control plane issue that is not transparent to Cloudera customers.

  • Environments running Cloudera DataFlow version 2.9.0-h2-b1 do not require this upgrade and will not have the upgrade banner.
  • Environments running Cloudera DataFlow version 2.9.0-h1-b1 or lower, will need to upgrade to 2.9.0-h3-b1.
  • Fixed issues can be found in 2.9.0 hotfix 2.

The 2.9.0-h2-b1release of Cloudera DataFlow fixes an issue preventing the running of test sessions, and an upgrade issue affecting single to Flexible Server migration in Azure environments.

Fixed Issues

  • Fixed an issue that was causing test sessions in Flow Designer to either take a long time to start, or not start at all.
  • Fixed an issue that was causing the 2.9.0 upgrade to fail for some customers running in Azure Environments.

The 2.9.0-h1-b1 release of Cloudera DataFlow fixes issues where the version 2.9.0 control plane had problems interacting with older workloads.

Fixed Issues

  • Fixed an issue where the Resources view handled workloads with version lower than 2.9.0 incorrectly.
  • Fixed an issue where Flow Designer became inaccessible in workloads with version lower than 2.9.0.
  • Fixed an issue where publishing a flow to the Catalog in a workload with version lower than 2.9.0 failed.

Cloudera Operational Database

The 1.46 release of the Cloudera Operational Database includes updates to the supported instance types and enhancements to the Operational Database UI.

A new CLI command to get the list of supported instance types
Cloudera Operational Database adds a new CLI command, list-supported-instance-types, allowing you to list the supported instance types for a combination of cloud platform, scale, and storage types. You can filter using the instance group and architecture as well.

The --custom-instance-types option in the create-database command allows you to pick and choose instance types for Cloudera Operational Database. However, it is essential that these instance types are included in the Cloudera Operational Database allowlist. The list-supported-instance-types command is useful to fetch the list of supported instance types.

Following is an example of the command:
cdp opdb list-supported-instance-types --cloud-platform AZURE --storage-type CLOUD_WITH_EPHEMERAL --scale-type MICRO --instance-group WORKER --architecture X86_64

For more information, see CDP CLI documentation.

Enhancements to the update-database command
A new CLI option --switch-instance-type is added to the update-database command to switch the instance types to custom instance types for the Cloudera Operational Database clusters after the database creation. You must have the COD_USE_CUSTOM_INSTANCE_TYPES entitlement to use this option.

If you plan to scale up the cluster vertically or change the existing instance type to any custom instance type, you can use the --switch-instance-type option. However, the custom instance types must be included in the allowlist by Cloudera Operational Database. Use the list-supported-instance-types command to fetch the list of supported instance types.

Following is an example of the usage:
cdp opdb update-database --environment-name <env_name> --database-name <database_name> --switch-instance-type instanceGroup=INSTANCEGROUP,instanceType=instancetype

For more information, see CDP CLI documentation.

Enhancements to the Cloudera Operational Database UI
Cloudera Operational Database UI provides two new UI options Configure Database and Configure Edge Nodes to configure the worker and edge node properties in your existing operational database.

You can access these menu items using the Actions menu item inside the database details page.

For more information, see Configuring a database.

GA support for the HBase REST server scaling
The feature to scale up the HBase REST server using the Apache HBase REST API is no longer a technical preview and is now generally available (GA).

For more information, see HBase REST server scaling in Cloudera Operational Database.

Cloudera Replication Manager

This release of the Replication Manager service introduces the following new features.

CDP CLI commands for HBase replication policies
You can create, activate, or delete HBase replication policies using the following CDP CLI commands:

  • activate-hbase-policy
  • create-hbase-policy
  • continue-hbase-setup
  • delete-hbase-policy
  • get-cluster-config
  • list-paired-hbase-clusters
  • suspend-hbase-policy
  • update-hbase-policy
  • verify-hbase-cluster-pair

For more information, see Creating HBase replication policy.