2022

Data Lake resizing

You can now scale up a light duty Data Lake to the medium duty form factor, which has greater resiliency than light duty and can service a larger number of clients. You can trigger the scale-up in the CDP UI or through the CDP CLI. For more information, see Data Lake resizing.

Data Lake backup and restore for GCP

Backing up and restoring a GCP Data Lake is now supported. For more information, see Backup and restore for the Data Lake.

Azure Reference Network Architecture

New conceptual overview of the CDP Public Cloud network architecture for Azure, its use cases, and personas who should be using it.

For more information, see Azure Reference Network Architecture.

Database Upgrade and default major version change

Newly deployed Data Lake and Data Hub clusters with Cloudera Runtime 7.2.7 or above are now configured to use a PostgreSQL version 11 database by default.

A new Database Upgrade capability is now available for existing Data Lake and Data Hub clusters. If you are running clusters on Cloudera Runtime version 7.2.6 or below, upgrade to a more recent version before performing the database upgrade.

The major version of the database used by Data Lake or Data Hub clusters is now also displayed on the Database page of the respective service.

Cloudera strongly recommends that the Database Upgrade is performed on all clusters running PostgreSQL version 10 before November 10, 2022.

For more information, see Upgrading database to Postgres 11

FreeIPA recipes and recipe type changes

You can register and attach recipes to run on a specific FreeIPA host group. For more information, see Recipes.

These changes will not affect existing recipe automation.

Validate and prepare for upgrade

Before you perform a Data Hub upgrade, you can run the new Validate and Prepare option to check for any configuration issues and begin the Cloudera Runtime parcel download and distribution. Using the validate and prepare option does not require downtime and makes the maintenance window for an upgrade shorter. For more information see Preparing for an upgrade.

AWS Hong Kong region

You can now register a CDP environment and create Data Hubs in the AWS Hong Kong Region (ap-east-1). See updated Supported AWS regions.

Changed permissions for managing proxies in CDP

This change has been introduced for new proxy registrations only; That is, proxies registered prior to this change continue to be managed by a PowerUser.

See updated Setting up a non-transparent proxy in CDP.

New documentation for CDP Public Cloud upgrade

FreeIPA scaling

You can resize your existing FreeIPA cluster via CDP CLI. Upscaling FreeIPA is recommended after performing Data Lake scaling. For more information, see Resize FreeIPA.

Support for Machine Learning in ap-1 and eu-1 regional Control Planes

New "Advanced Options" section in environment registration wizard

More options will be added to the "Advanced Options" in the future.

New option to delete attached volumes during Data Lake repair

When you initiate a repair from the Data Lake Hardware tab, you have the option to delete any volumes attached to the instance. For more information see Performing manual Data Lake repair.

Public Endpoint Access Gateway for Azure

Generate workload username based on email

By default, workload usernames are generated using the identity provider user ID. Alternatively, you can now generate workload usernames based on users' email addresses. This is useful in cases when the identity provider user ID is an opaque ID, like a uuid or employee ID, which gives equally opaque workload usernames. For more information, see Generating workload usernames based on email.

AWS Jakarta region

You can now register a CDP environment and create Data Hubs in the AWS Jakarta Region (ap-southeast-3). See updated Supported AWS regions.

Support for Operational Database in ap-1 and eu-1 regional Control Planes

Restricting all Cloudera SSO access

For added security, you can now restrict all Cloudera SSO access (including account administrator access) by contacting Cloudera Support and they can disable or enable the "Cloudera SSO All Login Enabled" setting for the account. Previously, account administrator access could not be restricted. For more information, see Disabling the Cloudera SSO login.

Customer managed encryption keys on GCP

Customer managed encryption keys on AWS

By default, Data Lake and FreeIPA's Amazon Elastic Block Store (EBS) volumes and Relational Database Service (RDS) are encrypted using a default key from Amazon’s KMS, but you can optionally configure encryption using Customer Managed Keys (CMK). Data Hubs inherit environment's encryption key by default but you have an option to specify a different CMK during Data Hub creation. For more information, refer to Adding a customer managed encryption key to a CDP environment running on AWS.

Deploying CDP in multiple AWS availability zones

By default, CDP provisions Data Lake, FreeIPA, and Data Hubs in a single AWS availability zone (AZ), but you can optionally choose to deploy them across multiple availability zones (multi-AZ). It is possible to enable it either for all or some of these components. For more information, refer to Deploying CDP in multiple AWS availability zones.

SCIM for Azure AD

CDP supports SCIM with Microsoft Azure Active Directory (Azure AD). For more information, see Configure SCIM with Azure AD.

New permissions were added to the default cross-account AWS policy

The cross-account access IAM role that is used for the CDP credential has been changed to include a set of new permissions required for Cloudera Data Engineering (CDE), Cloudera DataFlow (CDF), and Cloudera Machine Learning (CML). The new AWS permissions are required to simplify the creation of the Kubernetes cluster on AWS. As a result of this change, all customers using or planning to use CDE, CDF, or CML in CDP Public Cloud on AWS must update their existing cross-account permissions to ensure that these three data services can be created, enabled, or updated.

{ 
"Effect": "Allow",
 "Action": [
 "ssm:DescribeParameters",
 "ssm:GetParameter",
 "ssm:GetParameters",
 "ssm:GetParameterHistory",
 "ssm:GetParametersByPath"
 ],
 "Resource": [
 "arn:aws:ssm:*:*:parameter/aws/service/eks/optimized-ami/*"
 ]
}

Data Lake scaling (Preview)

Data Lake scaling is the process of scaling up a light duty Data Lake to the medium duty form factor, which has greater resiliency than light duty and can service a larger number of clients. You can trigger the scale-up in the CDP UI or through the CDP CLI. See Data Lake scaling (Preview).

Cloudera Runtime 7.2.15

Support for Replication Manager in ap-1 and eu-1 regional Control Planes

Bring your own Azure private DNS zone

Extended upgrade version support for RAZ-enabled environments

Data Lake major/minor version upgrades for RAZ-enabled environments are now available for Runtime versions 7.2.10-7.2.12 to 7.2.14+.

Changed FreeIPA Azure VM type

The Azure VM type used for the FreeIPA server was changed from Standard_D3_v2 to Standard_DS3_v2 so that FreeIPA nodes can be encrypted at host. Standard_D3_v2 doesn't support encryption at host while Standard_DS3_v2 does. See updated Overview of Azure resources used by CDP.

Setting IDBroker mappings in a RAZ environment is disabled

If a CDP environment has RAZ enabled, setting IDBroker mappings is disabled during environment creation and when the environment is already running. If your environment has RAZ enabled, you should be using Ranger for authorizing user and group access to the S3 or ADLS Gen 2 cloud storage used by the Data Lake.

Azure Load Balancer

The Standard SKU Azure Load Balancer is used in multiple places in CDP Data Lakes and Data Hubs. It is used as a frontend for Knox in both Data Lakes and Data Hubs, and for Oozie HA in HA Data Hubs. See Azure Load Balancers in Data Lakes and Data Hubs.

Upgrading classic clusters from CCMv1 to CCMv2

You can now upgrade your CDH, HDP, or CDP Private Cloud Base clusters that were previously registered in CDP from CCMv1 to CCMv2. See Upgrading a classic cluster from CCMv1 to CCMv2.

Upgrading FreeIPA

To ensure that your FreeIPA nodes are running with the latest patches, you should periodically upgrade your FreeIPA cluster. CDP currently allows you to upgrade all FreeIPA clusters, updating OS-level security patches on the cluster nodes. See Upgrade FreeIPA.

Upgrading the Data Lake

Major/minor version upgrades of Cloudera Runtime and Cloudera Manager are generally available. Data Lake maintenance upgrades for RAZ-enabled environments versions 7.2.7+ are generally available. For more information see Data Lake upgrade.

New classic cluster roles

As part of the new authorization model released in 2021, CDP introduces a new account role and resource roles related to classic clusters:

For more information, see Enabling admin and user access to classic clusters.

Data Lake backup and restore options

Public certificate auto-renewal

Most public (Let's Encrypt-issued) certificates for Data Lake and Data Hub clusters will now auto-renew without intervention from a user. For more information, refer to Managing Certificates.

Data Lake recipes

Support for attaching/detaching recipes on a Data Lake cluster is now available through both the CDP UI and CDP CLI. For more information see Recipes.

Cloudera Runtime 7.2.14

Cloudera Runtime 7.2.14 is now available and can be used for registering an environment with a 7.2.14 Data Lake and creating Data Hub clusters. See Cloudera Runtime.

Resource list filtering

CDP users other than PowerUsers and CDP administrators can only list the resources that they are authorized to access. Prior to this change, all CDP users were able to list all resources, but if they tried to access a resource that they were not authorized to access, CDP would return an error.

Customer managed encryption keys on Azure

By default, local Data Lake, FreeIPA, and Data Hub disks attached to Azure VMs and the PostgreSQL server instance used by the Data Lake and Data Hubs are encrypted with server-side encryption (SSE) using Platform Managed Keys (PMK), but you can optionally configure SSE with Customer Managed Keys (CMK). For more information, refer to Adding a customer managed encryption key for Azure.

Support for CDW in ap-1 regional Control Plane

Cloudera Data Warehouse (CDW) is now supported in the ap-1 (Australia) regional Control Plane. To use CDW in this regional Control Planes, your CDP administrator must create a new environment.

Workload password policies

In order to bring your workload password complexity requirements in line with company policy, you can set your FreeIPA password policies via CDP web interface and CDP CLI. Password policies can be configured for length, complexity, expiration, and scope. For more information, refer to Configuring workload password policies.

Pull-based audit archiving

Pull-based audit archiving allows you to pull audit events for archiving purposes without any extra configuration beyond Control Plane API usage. For more information refer to Pull-based audit archiving.

Custom images and catalogs

If necessary, you can use a custom Runtime or FreeIPA image for compliance or security reasons. You can then use the CDP CLI to register a custom image catalog and set the custom image within the custom image catalog. For more information refer to Custom images and catalogs.

Support for CDW in eu-1 regional Control Plane

Cloudera Data Warehouse (CDW) is now supported in the eu-1 (Germany) regional Control Plane. To use CDW in one this Control Plane, your CDP administrator must create a new environment.