CDP Public Cloud: January 2021 Release Summary

Cloudera is pleased to announce the latest updates to CDP Public Cloud.

Highlights

• CDE adds support for Airflow DAGs, per-cluster ACLs, troubleshooting and supports EKS 1.18
• Data Visualization is now integrated with CDP SSO
• CML now supports Applied ML Prototypes and EKS 1.18
• Data Hub adds support for cloud hardware resiliency via placement groups (AWS) and availability sets (Azure)
• Two new modules added to CDP Public Cloud Administration training course
• [Tech Preview] SDX Medium Duty v2 is available on AWS; v2 offers full resiliency for UI and API access

New Or Updated Capabilities

Documentation

• Management Console

  • Configuring audit event archiving via UI (AWS)

  • Configuring audit event archiving via UI (Azure)

  • Manually creating the ADLS account used for storing images and copying the images

• Data Engineering

  • Job Swagger APIs published and available for reference to all users

  • Lightweight backup & restore of Virtual Cluster Jobs metadata

  • Resource overview

  • Enabling WXM integration, tags, and IP whitelisting

  • Troubleshooting tools - log files and deep analysis

Cloudera Data Engineering

• Support for Airflow DAGs

  • DE and ML practitioners can now define their own pipelines packaged as an Apache Airflow Python DAG. Currently supported CDP operators include running Spark jobs on CDE and Hive jobs on CDW.

  • An embedded Airflow UI within the job & job run details pages gives users a “deep link” to the specific Airflow DAG making it easier to access within the context of the job runs.

  • The Schedule page has been removed from the left panel of the virtual cluster jobs UI, and the full Airflow UI is now exposed through the Virtual Cluster details page.

• Improved service observability for service troubleshooting

  • Diagnostic bundles can now be collected through a new API end-point, which includes:

    • Cloud resource status: (EKS, RDS, EFS, ELB)
    • Helm status( helm version, helm ls -A)
    • Kubernetes status (deployments, pods, services, ingresses, config maps)

• Virtual Cluster user-based ACL

  • By default a Virtual Cluster is accessible to all DEUsers and DEAdmins, which includes the Jobs API, Airflow UI, along with any connections and credentials defined within Airflow.

  • Enabling access control will now limit access to the API and UIs of the Virtual Cluster to a subset of users - normal or machine users. Groups are not yet supported.

• Kubernetes support

  • CDE now supports EKS 1.18

Data Visualization

• CDP SSO integration

  • CDP Data Visualization applications running in Cloudera Machine Learning released in November 2020 or later can permit SSO authentication in admin settings.

• Usability improvements

  • Improvements and bug fixes for crosstab visuals, custom display formats, CSV/Excel downloads and dashboard import/export.

  • Refer to additional release notes for Nov and Dec

Cloudera Machine Learning

• Applied ML Prototypes

  • Applied ML Prototypes (AMPs) are now generally available inside the product, which provide end-to-end prototypes to help kickstart real customer use cases. AMPs effectively package pre-built models for data scientists to tailor for their own needs, as well as enable data scientists to learn how the different parts of CML work together.

• Kubernetes Support

  • CML AWS now supports K8s 1.18, keeping up with updated EKS support

SDX

• [Tech Preview] Updates to SDX HA

  • SDX HA with full resiliency for UI & API access (‘Medium Duty v2’) is available on AWS (only v1 is available on Azure)

  • In addition to the builtin client-side failover available in v1, v2 offers full resiliency for UI and API access

Data Hub

• Support for Cloud Hardware Redundancy

  • Guaranteed physical hardware redundancy via AWS Placement Groups and Azure Availability Sets, preventing cloud provider hardware failures from affecting any HA services in Data Hub

Management Console & Control Plane

• Classic-Clusters Support for Centos 6.x

  • This will support onboarding of legacy on prem CDH clusters running on older centos versions to CDP Public Cloud.

Education

• CDP Public Cloud Administration Training

  • Two new modules in our course on CDP Public Cloud administration are now available to customers with an Cloudera OnDemand library subscription. The first, Classic Clusters, demonstrates how to register and manage classic clusters with CDP Public Cloud, which are essential skills for hybrid cloud deployments. The second, Monitoring and Management, explains how administrators can use Cloudera Manager to operate and monitor clusters in a CDP environment, including how to view cluster health, metrics, utilization, events, logs, and reports.

Full release documentation including release notes for each service is available here (Service Name → Release Notes → What’s New).