2023

December 20, 2023

This release of the Management Console service introduces the following changes:

VHD images hosted in Azure Marketplace

Cloudera now publishes VHD images on Azure Marketplace for each minor Runtime release (for example, 7.2.17) and CDP uses these images by default during environment and Data Hub creation. In order for CDP to be able to load these images, customers need to accept Azure Marketplace terms and conditions either via CDP web UI (recommended) or Azure CLI. Note that RHEL 8 images on Azure are only available via Azure Marketplace.

In order to use this feature, you need to Grant the service principal additional Azure permissions:

  1. On the scope of your Azure subscription: ​​
    "Microsoft.MarketplaceOrdering/offertypes/publishers/offers/plans/agreements/write",
    "Microsoft.MarketplaceOrdering/offerTypes/publishers/offers/plans/agreements/read”
  2. On the scope of the CDP Azure resource group:
    “Microsoft.Resources/deployments/whatIf/action” 

For more information, see CDP images hosted in Azure Marketplace.

December 1, 2023

This release of the Management Console service introduces the following changes:

Additional permissions needed in the cross-account role

You should update the AWS cross-account role to include ec2:DescribeLaunchTemplateVersions policy action to allow CDP update AWS Launch Template UserData. CDP Public Cloud cluster operations such as Cluster Connectivity Manager version upgrades, proxy, and Salt credential management require updating the AWS Launch Template UserData. Without this capability, node repair, upgrade, and scale operations performed on the CDP Public Cloud environment, Data Lake (SDX), and Data Hub may fail.

November 1, 2023

This release of the Management Console service introduces the following changes:

Resize an existing Data Lake from single-AZ to multi-AZ

As part of the Data Lake resizing via CDP CLI, you can optionally resize a Data Lake from single-AZ to multi-AZ by adding the --multi-az flag to the Data Lake resize command. This is available via CDP CLI only. For more information, see Data lake scaling and Scaling the Data Lake through the CDP CLI.

October 30, 2023

This release of the Management Console service introduces the following changes:

Qatar Doha (me-central1) and KSA (me-central2) GCP region support

CDP adds support for launching environments, Data Lakes, and Data Hubs in the Qatar Doha (me-central1) and KSA (me-central2) GCP regions. See updated Supported GCP regions.

October 18, 2023

This release of the Management Console service introduces the following changes:

Elastic load balancer deletion protection on AWS

When Data Lakes and Data Hubs are created, CDP deploys load balancers on AWS for endpoint stability. With the release of this update, all newly created load balancers for Data Lakes and Data Hubs on AWS are configured with deletion protection enabled, provided that your cross-account policy has the required permissions.

Cloudera has updated the AWS cross-account policy definition with the additional elasticloadbalancing:ModifyLoadBalancerAttributes permission required to set and remove the load balancer deletion protection flag. If you would like to use this feature, please update your cross account policy on AWS by adding the elasticloadbalancing:ModifyLoadBalancerAttributes permission. If you are using the old policy definition, this new feature will not be available (that is, the deletion protection will not be set).

October 9, 2023

This release of the Management Console service introduces the following changes:

Resizing to a Enterprise Data Lake

You can now resize a light or medium duty Data Lake to an Enterprise Data Lake (EDL). Resizing to an EDL is available only on Runtime 7.2.17 and above. For more information, see Data Lake resizing.

October 5, 2023

This release of the Management Console service introduces the following changes:

Public Endpoint Access Gateway for GCP

You can enable Public Endpoint Access Gateway during GCP environment registration. For more information, see Public Endpoint Access Gateway.

September 20, 2023

This release of the Management Console service introduces the following changes:

Support for il-central-1 AWS region

The Tel Aviv (il-central-1) AWS region is now supported. You can register CDP environments and provision Data Hubs in that region. See updated Supported AWS regions.

September 6, 2023

This release of the Management Console service introduces the following changes:

Audit event archiving for GCP

You can now configure audit event archiving for Google Cloud Platform.

For more information, see GCP setup for audit archiving.

August 30, 2023

This release of the Management Console service introduces the following changes:

Data Lake database upgrade and default major version change

Newly-deployed Data Lake clusters on AWS or GCP with Cloudera Runtime 7.2.7 or above are now configured to use a PostgreSQL version 14 database by default.

Newly-deployed Data Lake clusters on Azure with Cloudera Runtime 7.2.7 or above will continue to use a PostgreSQL version 11 database by default.

The database for Data Lake clusters on AWS and GCP can now be upgraded to PostgreSQL version 14. If your AWS or GCP cluster requires an upgrade to PostgreSQL 14, you will receive a notification in the Management Console UI.

Cloudera strongly recommends that the database upgrade to PostgreSQL 14 for AWS and GCP clusters is performed on all clusters running PostgreSQL version 11 by November 9, 2023.

A database upgrade to PostgreSQL 14 for Azure Data Lakes will be available in the future. Any Data Lake clusters on Azure that require a database upgrade will be upgraded from PostgreSQL 10 to PostgreSQL 11.

For more information, see Upgrading Data Lake/Data Hub database

August 23, 2023

This release of the Management Console service introduces the following changes:

Configure proxy for TLS interception and Deep Packet Inspection

After setting up a web proxy server in CDP, you can further configure it to perform TLS interception and Deep Packet Inspection (DPI). For instructions, see Setting up a web proxy for TLS inspection.

Setting a default identity provider in CDP

You can optionally set a default identity provider (IdP) in CDP. If you do so, CDP will use the default IdP instead of the oldest IdP. For instructions, see Setting a default identity provider in CDP.

July 20, 2023

This release of the Management Console service introduces the following changes:

Enterprise Data Lakes

A new Data Lake shape called the enterprise Data Lake is now available for new deployments of Cloudera Runtime version 7.2.17. The enterprise Data Lake is a redefined version of medium duty Data Lakes that still offer failure resilience, but utilize resources and allocate memory more efficiently than a medium duty Data Lake at the same cost. For more information see Data Lake scale.

Medium duty Data Lakes deprecated

Medium duty Data Lakes are deprecated as of Runtime 7.2.17. You can upgrade a medium duty Data Lake from 7.2.16 to 7.2.17, but will not be able to upgrade it further. You can create a new 7.2.17 medium duty Data Lake through the CDP CLI, but Cloudera recommends using the enterprise Data Lake for new deployments. The ability to create medium duty Data Lakes will be removed from both the UI and CLI from 7.2.18.

July 19, 2023

This release of the Management Console service introduces the following changes:

CCM enabled by default in the UI

When you register a new AWS, Azure, or GCP environment in CDP via web interface, Cluster Connectivity Manager (CCM) is enabled by default and the option to disable it has been removed from the web interface. Also, CCM is now enabled by default in CDP CLI.

CCMv2 upgrade

CCMv2 upgrade is available for all customers. If your existing CDP environment was created with CCMv1, you will see a notification in your environment details to upgrade to CCMv2. See Upgrading from CCMv1 to CCMv2.

June 28, 2023

This release of the Management Console service introduces the following changes:

Cloudera Runtime 7.2.17

Cloudera Runtime 7.2.17 is now available and can be used for registering an environment with a 7.2.17 Data Lake and creating Data Hub clusters. For more information about the new Runtime version, see Cloudera Runtime. If you need to upgrade your existing CDP environment, refer to Data Lake upgrade and Data Hub upgrade documentation.

Rolling Data Lake upgrades

With the release of Cloudera Runtime 7.2.17, you can now upgrade your Data Lake in a rolling manner. The rolling upgrade allows you to upgrade the Data Lake Runtime and OS without stopping attached Data Hubs or Data Services. This allows workloads to continue running during the Data Lake upgrade operation.

GCS fine-grained access control (Preview)

You can now register a CDP environment on GCP with RAZ enabled to use fine-grained access policies and audit capabilities available in Apache Ranger. See GCS Fine-Grained Access Control.

Cross-version compatibility for Data Lake backup and restore

You can take a backup of a Data Lake that runs one version of Cloudera Runtime and restore the backup to a Data Lake that runs a different version of Runtime. The backup version must be an earlier/lower version Runtime than the Data Lake that you are restoring to. Version limitations apply and a Ranger/HMS schema upgrade may be required. See Cross-version support for Data Lake backup and restore for more details.

Cross-version compatibility for Data Lakes and Data Hubs

Backward compatibility between Data Lakes and Data Hubs has been introduced with Cloudera Runtime 7.2.17. It is no longer required that you perform Data Hub upgrades in lock-step with the Data Lake upgrade. If your Data Hub is on Runtime version 7.2.16 or later, it is compatible with a Data Lake on a newer Runtime version (7.2.17+). You can independently upgrade your Data Hubs at a later time if you choose to, though it is not required.

AWS Middle East UAE region

You can now register a CDP environment and create Data Hubs in the AWS Middle East UAE region (me-central-1). See updated Supported AWS regions.

Data Lake upgrade validations for Python dependency

If you are planning to upgrade the Runtime version in your existing Data Lake clusters to 7.2.17 or higher versions, you may be required to perform an additional step before upgrading.

You can verify whether your cluster is impacted by navigating to the Upgrade tab of your Data Lake cluster. If there is a warning message about missing prerequisites on the Upgrade page, you need to perform an additional upgrade step before moving to the 7.2.17 Cloudera Runtime version. The required steps may be different depending on your current Runtime version. See Data Lake upgrade for more details.

June 13, 2023

This release of the Management Console service introduces the following changes:

Zone selection within a GCP region

During GCP environment registration you can now select a zone within the selected GCP region. For example, if you selected the us-west1 region, you can select one of its three zones: us-west1-a, us-west1-b, or us-west1-c.

CDP credit consumption and usage insights

CDP includes a user interface that allows you to monitor your credit consumption and download your consumption records. See CDP credit consumption and usage insights.

April 19, 2023

This release of the Management Console service introduces the following changes:

FreeIPA and Data Lake instance type selection

You can now select the FreeIPA and Data Lake instance types when you register an environment. For more information see Register an AWS environment from CDP UI, Register an Azure environment from CDP UI, and Register a GCP environment from CDP UI.

AWS GP3 support for attached storage disks

AWS Data Lake and FreeIPA instances now support GP3 (SSD) volume types for attached storage. GP3 volumes allow you to increase performance (independently provisioning IOPS and throughput) without increasing storage size. GP3 volumes will deliver similar performance as GP2 volumes at a lower cost. GP3 is now the default attached storage type for instances that previously used GP2 storage.

March 29, 2023

This release of the Management Console service introduces the following changes:

FreeIPA instance type on GCP

Default FreeIPA instance type on GCP changed from n1-standard-2 to e2-standard-2.

January 11, 2023

This release of the Management Console service introduces the following changes:

Cloudera Runtime 7.2.16

Cloudera Runtime 7.2.16 is now available and can be used for registering an environment with a 7.2.16 Data Lake and creating Data Hub clusters. For more information about the new Runtime version, see Cloudera Runtime. If you need to upgrade your existing CDP environment, refer to Data Lake upgrade and Data Hub upgrade documentation.

Instance type selection for Data Lake and FreeIPA on AWS

If necessary, you can now select a larger or smaller instance type for a Data Lake or FreeIPA after the environment has been deployed. This is only supported on AWS. For more information see Vertically scaling instances types - Data Lake and Vertically scale FreeIPA instances.