Cloudera Docs

December 08, 2022

This release (2.3.0-b347) of Cloudera DataFlow (CDF) on CDP Public Cloud introduces the technical preview of Flow Designer, adds several new ReadyFlows, improves upgrade reliability and fixes issues with stability for clusters with high utilization.

New features and changes

  • Flow Designer [Technical Preview]

    Developers can now build new data flows from scratch using the integrated Designer. Flow Drafts can be tested using Test Sessions before they are published to the Catalog. Once they are in the catalog, users can create Flow Deployments using the Deployment Wizard or CLI.

  • Added support for certificate renewal of inbound connection endpoints

  • Six new ReadyFlows have been added to the ReadyFlow Gallery:

    • Airtable to S3/ADLS

    • Box to S3/ADLS

    • Dropbox to S3/ADLS

    • Google Drive to S3/ADLS

    • MQTT to Kafka

    • Salesforce to S3/ADLS

  • Apache Iceberg Integration [Technical Preview]

    Developers can now use the PutIceberg processor to write data to Apache Iceberg in CDP.

  • DataFlow is now also available in the EU and APAC regional CDP control planes.

  • Updated to support using Kubernetes 1.23 EKS and AKS clusters.

    If you use restricted IAM roles in your EKS clusters, this impacts the upgrade procedure. For more information, see the CDF Service Upgrade documentation.

  • General improvements to DataFlow upgrade reliability to allow for improved success rates.

  • Includes new NiFi version 1.18.0.2.3.7.0 for flow development and flow deployment.

  • Updated DataFlow Function binaries to use NiFi version 1.18.0.

Fixed issues

  • CDPDFX-5884 - Internal Logging Improvements - fixed logging issues sent to S3 (AWS) and Azure Blob Storage (Azure)
  • CDPDFX-6308 / CDPDFX-6323 - updated legacy inbound connection endpoints to make them compatible with new certificate renewal capability
  • CDPDFX-6305 - Fixed issue with inbound connection endpoints created prior to DFX 2.2.0 preventing upgrade
  • CDPDFX-5689 - Expanded character limit on deployment parameters to allow for parameters of up to 100000 characters.
  • CDPDFX-5942 - Fixed a rare issue where a cluster update request can fail due to cloud API latency leading to timeouts.
  • CDPDFX-6334 - Fixed an issue where a large deployment count with active metrics unavailable alerts can lead to a deadlock which prevents upgrade
  • CDPDFX-6345 - Fixed issue which causes existing deployments using assets to no longer allow changes to those assets after upgrade due to a certificate mismatch
  • CDPDFX-6364 - Fixed CLI issue where some listen components are ignored when creating deployments with multiple listen components
  • CDPDFX-6378 - Fixed issue where a post-upgrade cleanup effort can fail with a warning if a customer’s outbound network restrictions do not allow retrieval of a liquibase schema file.
  • CDPDFX-5731 - Cert-manager OOM - increased memory limits for cert-manager to ensure certificates can still be issued as deployment numbers increase
  • CDPDFX-6371 - Correct nifi.provenance.repository.debug.frequency to a parseable default value (default 1000000)
  • CDPDFX-6309 - ZK-Operator OOM - prevent operator from running OOM due to it scan/filtering of multiple secrets
  • CDPDFX-6118 - Suppress deprecation logging in NiFi
  • CDPDFX-6387 - Increased cpu limit for cer-manager-webhook container
  • CDPDFX-6235 - Fix errors seen during NiFi autoscaling due to inconsistent nifi identity configmap updates
  • CDPDFX-6230 - Fix to prevent metering alert from firing when DataFlow service not fully enabled