2021

Fine-grained Access Control for ADLS Gen2 and S3

The fine-grained access control for ADLS Gen2 and S3 cloud storage via the Ranger Authorization Service (RAZ) enables Amazon S3 and ADLS Gen2 users to control access per user and per directory in cloud storage. By specifying Apache Ranger policies for cloud storage, admins can provide home directories and audit capabilities similar to those used with HDFS files in an on-premises or IaaS deployment.

For more information and setup steps refer to:

• Fine grained access control for AWS
• Fine grained access control for Azure

AWS Milan region is supported for CDW

Cloudera Data Warehouse (CDW) introduces support for the eu-south-1 (Milan) AWS region. See updated AWS regions supported by CDP.

No-proxy option for non-transparent proxies

When you set up a non-transparent proxy server, you now have the option of configuring specific IP addresses, domains, or subdomains to bypass the proxy. For more information, see Using a non-transparent proxy.

CDW diagnostics collection

You can trigger a diagnostics bundle collection for Cloudera Data Warehouse (CDW). See updated Send a diagnostic bundle to Cloudera Support.

Updated GCP provisioning credential's permissions

compute.regionHealthChecks.useReadOnly

If your GCP provisioning credential uses a custom IAM role with granular permissions, you should update it to include this permission.

See updated Service account for the provisioning credential.

Updated Azure provisioning credential's permissions

"Microsoft.Network/loadBalancers/backendAddressPools/join/action",
"Microsoft.Network/loadBalancers/delete",
"Microsoft.Network/loadBalancers/read",
"Microsoft.Network/loadBalancers/write",

If you have created a custom role for the CDP provisioning credential, you should update your application registration on Azure, assigning these additional permissions. If you have assigned the built-in Contributor role instead of granular permissions, you do not need to take any action.

Documentation has been updated. See Prerequisites for the provisioning credential.

FreeIPA HA for GCP environments

FreeIPA HA is now supported and used by default for all newly created GCP environments.

Cluster Connectivity Manager v2 (CCMv2)

CCMv2 replaces CCMv1. While CCMv1 establishes and uses a tunnel based on the SSH protocol, with CCMv2 the connection is via HTTPS. All new environments created with Runtime 7.2.6 or newer after enabling CCMv2 on your tenant use CCMv2. Existing environments and new environments created with Runtime older than 7.2.6 continue to use CCMv1. All newly registered classic clusters use CCMv2, but previously registered classic clusters continue to use CCMv1. If your CDP tenant has not been granted the CDP_CCM_V2 entitlement yet, it continues to use CCMv1.

The steps to register an environment with CCMv2 are similar to CCMv1 configuration steps. The main differences are:

• If you are deploying in an environment with restricted outbound network access, port 443 needs to be open and new destinations need to be added to the allow list.
• If you are registering a classic cluster, the steps have changed.

For more information, see Cluster Connectivity Manager.

Medium duty Data Lakes for GCP

Medium duty Data Lakes for GCP have added an additional gateway node to provide failure resilience for UI and API clients. Load-balanced UI and API access are now available without interruption. For more information see Data Lake scale.

Cloudera Runtime 7.2.12

Cloudera Runtime 7.2.12 is now available and can be used for registering an environment with a 7.2.12 Data Lake and creating Data Hub clusters. See Cloudera Runtime.

New GCP permissions for provisioning credential

The list of permissions for the provisioning credential's service account has been updated to include new permission required for load balancing between HA components of the Data Lake. If you are running CDP in GCP, you should update the provisioning credential's service account to include either the Compute Load Balancer Admin (roles/compute.loadBalancerAdmin) IAM role or the following granular permissions:

• compute.addresses.create
• compute.addresses.delete
• compute.addresses.get
• compute.addresses.use
• compute.instanceGroups.create
• compute.instanceGroups.delete
• compute.instanceGroups.get
• compute.instanceGroups.list
• compute.instanceGroups.update
• compute.instanceGroups.use
• compute.forwardingRules.create
• compute.forwardingRules.delete
• compute.forwardingRules.get
• compute.forwardingRules.list
• compute.forwardingRules.setLabels
• compute.forwardingRules.update
• compute.forwardingRules.use
• compute.regionBackendServices.create
• compute.regionBackendServices.delete
• compute.regionBackendServices.get
• compute.regionBackendServices.list
• compute.regionBackendServices.update
• compute.regionBackendServices.use
• compute.regionHealthChecks.create
• compute.regionHealthChecks.delete
• compute.regionHealthChecks.get
• compute.regionHealthChecks.list
• compute.regionHealthChecks.update
• compute.regionHealthChecks.use

See updated Permissions for the provisioning credential's service account.

New authorization model

CDP introduces a new authorization model. The following table summarizes new, changed, and deprecated roles. The roles that are not mentioned in this table are unchanged.

Account roles

Resource roles

Steps for assigning roles

• The steps for assigning account roles and managing access to environments are unchanged.
• The steps for managing access to Data Hubs, shared resources (cluster templates, credentials, image catalogs, and recipes), and classic clusters are similar to the steps for managing access to environments: You can use the Manage access option from the resource details page.

Updated documentation

• For updated information about all built-in roles in CDP, refer to Understanding account-level roles and resource roles.
• For updated instructions for how to manage access to resources, refer to Assigning a resource role to a user and Assigning a resource role to a group.
• Other new and updated documentation:
  • Enabling admin and user access to environments
  • Enabling admin and user access to Data Hubs

Dots are now supported in group names

Improved AWS cloud storage setup documentation

AWS cloud storage setup documentation has been improved to include exact steps for creating the required S3 bucket, IAM policies, and IAM roles. See Minimal setup for cloud storage and Onboarding CDP users and groups for cloud storage .

Cloudera Runtime 7.2.11

Cloudera Runtime 7.2.11 is now available and can be used for registering an environment with a 7.2.11 Data Lake and creating Data Hub clusters. See Cloudera Runtime.

Specifying multiple existing AWS security groups

When using your existing security groups for registering an AWS environment in CDP via CDP CLI, you can provide a comma-separated list of multiple security groups for both Knox (securityGroupIdForKnox) and Default (defaultSecurityGroupId). This is a CLI-only feature.

Specifying multiple GCP shared subnets

When using an existing shared VPC for registering a GCP environment in CDP via CDP web interface or CLI, you can specify multiple shared subnets.

Support for Bahrain (me-south-1) AWS region

Registering an environment and provisioning Data Hubs is now supported in the Bahrain (me-south-1) AWS region. See Supported AWS regions.

Updated outbound network access destinations

If you are using Machine Learning, Data Engineering, or DataFlow data services and have restricted egress access, starting on September 7, 2021, you need to add the following new endpoints to your egress rules:

• *.s3.{eu-west-1, us-west-2, ap-southeast-1}.amazonaws.com
• s3-r-w.{eu-west-1, us-west-2, ap-southeast-1}.amazonaws.com
• *.execute-api.{eu-west-1, us-west-2, ap-southeast-1}.amazonaws.com

The region selected should be the region that is geographically closest to where the environment is deployed.

Customers operating in outbound restricted networks will be unable to download docker images, which will impact creating new clusters. Existing environments deployed in outbound restricted networks may experience operational issues, including limited ability to start, scale and repair the data service clusters.

New Shared Resources navigation menu item

Management options for provisioning credentials, proxies, and Data Hub cluster templates, recipes, and image catalogs can now be easily accessed from the new Shared Resources item in the navigation menu.

Send a diagnostic bundle to Cloudera support

CDP introduces a web interface for sending a diagnostic bundle to Cloudera support. Currently diagnostics can be collected for Data Lake, FreeIPA, and Data Hub. See Send a diagnostic bundle to Cloudera support.

New permission for GCP Logger service account

In addition to the previously documented permissions, if you would like to use a bucket path (gs://<bucket>/<path>) instead of a bucket (gs://<bucket>) for the Logs or Backups bucket, you should assign the storage.objects.list permission to the custom role. See Minimum setup for cloud storage.

"Create public IPs" is disabled with CCM

The Create public IPs option available on the UI during Azure and GCP environment registration is now disabled by default when CCM is enabled.

Updated quick starts

The AWS, Azure, and GCP quick starts have been updated to include the optional FreeIPA Backup Location Base introduced in a recent release.

FreeIPA HA repair

FreeIPA HA repair is now available for all newly created AWS and Azure environments in CDP. When running in high-availability mode, the identity management system runs multiple instances of FreeIPA on separate hosts. In case of failure, you can now repair failed hosts using the CDP command-line within one week of a node failing. For more information, see Repair a FreeIPA instance.

"Don't create public IPs" option was renamed

The Don't create public IPs option available during Azure and GCP environment registration was renamed to Create public IPs and is enabled by default.

FreeIPA backup location

FreeIPA HA enabled by default for new AWS and Azure environments

FreeIPA HA is now enabled for all newly created AWS and Azure environments in CDP. The number of nodes used for the FreeIPA server depends on the Data Lake scale selected: light duty uses 2 nodes and medium duty uses 3. The FreeIPA HA toggle button has been removed from the environment registration UI, but, if needed, it is possible to customize FreeIPA node count when registering an AWS or Azure environment via CDP CLI. For more information, see Managing FreeIPA.

Cloudera Runtime 7.2.10

Cloudera Runtime 7.2.10 is now available and can be used for registering an environment with a 7.2.10 Data Lake and creating Data Hub clusters. See Cloudera Runtime.

S3Guard removal

S3Guard is no longer used with newly registered AWS environments using Runtime version 7.2.2 or newer. Consequently, the ''Enable S3Guard" environment registration option has been removed and there is no need to create a DynamoDB table for your environment when planing to use Runtime version 7.2.2 or newer. Environments created prior to this change continue to use S3Guard.

Updated GCP Quick Start

GCP Quick Start has been updated to include environment registration steps using CDP web interface. See GCP Quick Start.

Updated CCM troubleshooting documentation

CCM troubleshooting documentation has been updated to include information on common cases when connection via CCM fails and steps for collecting information from clusters. See Troubleshooting CCM.

Cloudera Runtime 7.2.9

Cloudera Runtime 7.2.9 is now available and can be used for registering an environment with a 7.2.9 Data Lake and creating Data Hub clusters. See Cloudera Runtime.

Registering CDP Private Cloud Base clusters in CDP Public Cloud

For documentation, see Adding a CDP Private Cloud Base cluster.

CDP Public Cloud onboarding documentation was moved

This CDP Public Cloud library is accessible via the Get Started link on the https://docs.cloudera.com homepage or via the https://docs.cloudera.com/cdp-public-cloud/cloud/index.html link:

• https://docs.cloudera.com/cdp/latest/requirements-aws/index.html
• https://docs.cloudera.com/cdp/latest/requirements-azure/index.html
• https://docs.cloudera.com/cdp/latest/requirements-gcp/index.html

• https://docs.cloudera.com/cdp/latest/security-overview/index.html

URL redirects were added temporarily; They will eventually be removed, so make sure to update your bookmarks.

This change was made in effort to make CDP Public Cloud onboarding documentation easier to find. The previous location of this content (in the Management Console library) was unintuitive to many users.

AWS/Azure/GCP planning documentation was consolidated

The AWS/Azure/GCP requirements content was consolidated in one place in the CDP Public Cloud library mentioned above.

This change was made in effort to consolidate all documentation related to cloud provider requirements in one place. Previously, the documentation was scattered and users had to click on many links in order to find content.

Cluster Definitions page was moved to environment details

The Cluster Definitions page that used to be available in the Shared Resources section was removed. Instead, you can access all cluster definitions related to a specific environment from the Cluster Definitions tab available in the environment's details. You can save new cluster definitions using the Save As New Definition option available from the Create Data Hub wizard or from CDP CLI using the cdp datahub create-cluster-definition command.

Ranger Audit environment parameter was moved to Data Access section

The option to specify the Ranger Audit role (AWS) managed identity (Azure) or service account (GCP) during environment registration was moved from the Logs - Storage and Audit section to the Data Access section. Consequently, these sections were renamed to Logs and Data Access and Audit.

You can select specific nodes to repair within a Data Lake host group

From the Hardware tab of the Data Lake details, you can click the Repair icon to select specific nodes within a host group to repair.

Updated IAM policy for the provisioning credential for AWS

cloudformation:UpdateStack
cloudformation:ListStackResources
elasticloadbalancing:DescribeLoadBalancers
elasticloadbalancing:DescribeTargetHealth
elasticloadbalancing:RegisterTargets
elasticloadbalancing:DeregisterTargets

If you are using a restricted IAM policy for your provisioning credential, you must add these additional permissions.

GCP quick start

GCP quick start is now available, allowing you to quickly set up a CDP environment. See GCP quick start.

Google Cloud Platform (GCP) support

Medium duty Data Lakes on Microsoft Azure

The medium duty Data Lake configuration is now available for Microsoft Azure. Light duty is still used by default, but you can change this when registering an environment from CDP user interface or when creating a Data Lake from CDP CLI using the --scale MEDIUM_DUTY_HA option. For information about available configurations, see Data Lake scale.

Cloudera Runtime 7.2.8

Cloudera Runtime 7.2.8 is now available and can be used for registering an environment with a 7.2.8 Data Lake and creating Data Hub clusters. See Cloudera Runtime.

Medium duty Data Lake for AWS

The medium duty Data Lake configuration is now available for AWS. Light duty is still used by default, but you can change this when registering an environment from CDP user interface or when creating a Data Lake from CDP CLI using the --scale MEDIUM_DUTY_HA option. For information about available configurations, see Data Lake scale.

New supported AWS regions

The eu-south-1 (Europe - Milan) and af-south-1 (Africa - Cape Town) regions are now available as technical preview for creating Data Hub clusters. See updated Supported AWS regions.

Cluster definitions location

Cluster definitions that can be used with a given environment are now listed in the details of that environment. To view them navigate your environment and access the Cluster Definitions tab.

Single existing Azure resource group

The option to use your existing Azure resource group is now available in CDP. This allows you to have your credential's role definition scoped to that particular resource group instead of the whole subscription. The option to "Select Resource Group" is available on the UI on the Region, Networking and Security page of the register environment wizard. The corresponding CLI JSON parameter is resourceGroupName and the cdp environments create-azure-environment CLI option to enable is --resource-group-name <value>. If these parameters are not present, CDP defaults to creating new resource groups. See updated Azure Permissions and Resource group in Azure Requirements documentation, and updated Register an Azure environment.

Private endpoints for Azure database for PostgreSQL

The option to create private endpoints instead of service endpoints for Azure Database for PostgreSQL is now available when registering an Azure environment in CDP. The option to "Create Private Endpoints" is available on the UI in the "Network" section of the register environment wizard. The corresponding CLI JSON parameter is createPrivateEndpoints and the cdp environments create-azure-environment CLI option to enable private endpoints is --create-private-endpoints. The option is disabled by default. It can only be used with the single resource group feature and can only be enabled on subnets that have Azure network policies turned off. See Private endpoint for PostgreSQL.

Public Endpoint Access Gateway for AWS

During AWS environment registration, you can optionally enable Public Endpoint Access Gateway, which provides secure connectivity to UIs and APIs in Data Lake and Data Hub clusters deployed using private networking, allowing users to access these resources without complex changes to their networking or creating direct connections to cloud provider networks. See Public Endpoint Access Gateway.

Generate CLI command from the UI

Disable cloud storage logging for an existing environment

By default, CDP sends collected service logs from VM nodes to the cloud storage that you provided for logs during environment registration. You can disable cloud storage logging for a specific environment, by navigating to environment details > Summary > Telemetry and turning off "Enable Cloud Storage Logging". Disabling this option will affect only newly created Data Hub clusters in that environment. See Enabling environment telemetry>Disable cloud storage logging for an existing environment.

Obtain your CDP tenant ID

You can now obtain your CDP tenant ID from CDP web interface,. See Obtain your CDP tenant ID.

Cloudera Runtime 7.2.7

Cloudera Runtime 7.2.7 is now available and can be used for registering an environment with a 7.2.7 Data Lake and creating Data Hub clusters. See Cloudera Runtime.

User delete

CDP administrators now have the ability to delete users in CDP through both the user interface and the CLI. Deleting a user removes all access keys and SSH keys associated with the user, and unassigns all roles and resource roles assigned to the user. The user is also removed from all groups that they belong to. For more information, refer to Deleting users and machine users.

FreeIPA HA

CDP administrators can configure your CDP environment to run FreeIPA in high-availability mode. By default, creating an environment configures a single instance of FreeIPA on its own host, but you can explicitly enable FreeIPA HA during environment registration via CPD web UI or CLI. For more information, refer to Managing FreeIPA.

Interactive login for CDP CLI and CDP SDK

If you would prefer that user access to the CLI/SDK is shorter-lived, you can use the "interactive" method of logging into the CDP CLI/SDK. By default, this login method grants a 12-hour access key to the CLI/SDK. The access key will time out after one hour of inactivity. The interactive method integrates with any SAML-compliant external identity provider. For more information, refer to Logging into the CDP CLI/SDK.

Anonymization rules

CDP includes a set of default anonymization rules and allows CDP administrators to define custom anonymization rules in order to remove sensitive information from CDP logs. For more information, refer to Defining anonymization rules for CDP logs.

Changes to delete machine user behavior

Deleting a machine user removes all access keys and SSH keys associated with the machine user, and unassigns all roles and resource roles assigned to the machine user. The machine user is also removed from all groups that they belong to. Previously, these steps had to be performed manually prior to machine user deletion. It takes around 2 minutes to fully delete a machine user in CDP. During that time you will not be able to recreate the machine user (that is, for 2 minutes you will not be able to create a machine user with the same machine user name).

Group name length limit

CDP user management framework supports group names of up to 64 characters. Previously up to 32 characters were supported.

Identity provider configuration improvements

The user interface and the overall flow of the identity provider configuration in CDP was improved for better usability.

New CDP SAML Service Provider certificate

The current CDP SAML Service Provider certificate is expiring on March 8, 2021 at 18:05:49 GMT. A replacement certificate is available for any customer whose identity provider will verify the CDP SAML service provider certificate. You can obtain the certificate from this document or by logging it to CDP web interface, navigating to > User Management > Identity Providers, clicking on your identity provider, and the last field "CDP SAML Service Provider Metadata" now contains 2 certificates: the one that expires on March 8, 2021 and the new one. Please consult your identity provider's documentation for how to update service provider certificates. CDP will start using the new certificate for SAML starting March 8, 2021.

Here is the new CDP SAML Service Provider certificate:

-----BEGIN CERTIFICATE-----
MIIEKTCCAxGgAwIBAgIUF7LjOby+L8dcCVzWN4ChnTtybiowDQYJKoZIhvcNAQEL
BQAwgaMxCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJDQTEWMBQGA1UEBwwNU2FuIEZy
YW5jaXNjbzEVMBMGA1UECgwMQ2xvdWRlcmEgSW5jMRAwDgYDVQQLDAdDRFAgSUFN
MSEwHwYDVQQDDBhjb25zb2xlLmNkcC5jbG91ZGVyYS5jb20xIzAhBgkqhkiG9w0B
CQEWFHN1cHBvcnRAY2xvdWRlcmEuY29tMB4XDTIxMDIyMzE5NDgxMVoXDTI0MDIy
ODE5NDgxMVowgaMxCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJDQTEWMBQGA1UEBwwN
U2FuIEZyYW5jaXNjbzEVMBMGA1UECgwMQ2xvdWRlcmEgSW5jMRAwDgYDVQQLDAdD
RFAgSUFNMSEwHwYDVQQDDBhjb25zb2xlLmNkcC5jbG91ZGVyYS5jb20xIzAhBgkq
hkiG9w0BCQEWFHN1cHBvcnRAY2xvdWRlcmEuY29tMIIBIjANBgkqhkiG9w0BAQEF
AAOCAQ8AMIIBCgKCAQEAszIxvwxxsAE4PqNLfZ2+4zfYI9UpiiePEOKJuL1Q8Mbh
ArA53EmZradpYNIQ54a3vGQNeEoi782gcp/JbzLTY0AESnKXzpPXOhX8qMWytrcL
QKmSW/eVbZsVEYnyf1wFxtpOcLbHfYB12W1ScD3FKr5BUns6bRCclfiFW1Ei5XLQ
yzgSGdKXSvB/8izRr4yyyDT2IX8PelHbONiIKb6OTuuHPwo259RMjZZd2pwMurif
JUGBckwYPh7Dkmiw9mTXVSD5fdSP1HvP2RTuPqmkTSgJRwdJD4G6wF1NFOQwItIr
7vf6OzPZJM6A2JCN8RQApMnYyNgT75wWtCNOF8F2cQIDAQABo1MwUTAdBgNVHQ4E
FgQUGfVSdXrVb3JsJy5nf4OYp2sJn8IwHwYDVR0jBBgwFoAUGfVSdXrVb3JsJy5n
f4OYp2sJn8IwDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAkNxk
+X2sCbXAIhSUNKUYQEM++ZDSnWzMgdavNeVUzWgTfGdwvDolFzvqU66wiQ8kedK0
qLW6gRZkG+GJUq5vY93pfNSQ5C4P9hhFqpd6tfHme7uHlZCtZh/wjOeYoOpgr0eI
qtXxg6U6+6qLqzBi/9Zdc0sLZFNbjQLEFkNHoU7rFODcnLNHemngw+ui2rofsBhK
F9Zcqiy91mmCto6OrQMAkXyfrU40S8+Yr9s+wnJEmNIkVN9mfH0TfRJNEvHcuvZ+
WHc4HD/Vu0sL/APPADfLh158MYb9gUNXtE12PxjGYCj4RsFt0/Fbju9mGl+W/n69
qaRFxZmubutaQ1WCzw==
-----END CERTIFICATE-----

CDP CLI reference

CDP CLI reference documentation is now available at https://cloudera.github.io/cdp-dev-docs/cli-docs/.

Documentation for configure lifecycle management for logs on AWS and Azur

To avoid unnecessary costs related to Amazon S3 pr ADLS Gen2 cloud storage, you should create lifecycle management rules for your cloud storage location used by CDP for storing logs so that these logs get deleted once they are no longer useful. See Configure lifecycle management for logs on AWS and Configure lifecycle management for logs on Azure.

Consolidated documentation for restricting admin and end user access for CDP services

Consolidated documentation for restricting admin and end user access for CDP services is now available. Previously these options were only covered in documentation related to specific CDP workload services. See Restricting access for CDP services that create their own security groups on AWS and Restricting access for CDP services that create their own security groups on Azure.

Updated AWS and Azure requirements documentation

AWS and Azure requirements documentation was updated to include more requirements related to Data Engineering, Data Warehouse, and Machine Learning. These requirements were previously only documented in service-specific docs. See updated AWS Requirements and Azure Requirements.

Control Plane audit archiving

You can configure Control Plane audit archiving from CDP web interface. Previously, this feature was only available via CDP CLI. For updated documentation, refer to AWS setup for audit archiving and Azure setup for audit archiving.

New documentation for pre-creating ADLS Gen2 account for storing OS images

CDP uses an ADLS Gen2 storage account for storing images used for VMs. By default, CDP creates this account during environment registration, but you can optionally pre-create it. If needed, you can also copy the VHD files and create image resources manually. For instructions on how to do this, refer to ADLS Gen2 account for storing images.

Python 3.6 is required for CDP CLI

Starting in January 2021, the CDP CLI requires Python 3.6 for execution. Python 2 reached end of life on January 1, 2020 and is no longer receiving updates. While your existing CDP CLI installation running on Python 2 will keep working, new features and bug fixes are available only for CDP CLI installations running on Python 3.6. To check your Python version, use:

python -V

Once you have Python 3, install CDP client by following the usual CDP client installation instructions.