October 5, 2023

This release (2.6.0-b311) of Cloudera DataFlow (CDF) on CDP Public Cloud introduces NiFi 1.23, vertical scaling, several cost optimization features and the ability to collect Diagnostic Bundles through the UI.

What's new

  • Added support for downloading NiFi application logs directly from CDF for deployments and test sessions.

  • Added support for suspend and resume CDF Deployments. Suspending a Deployment temporarily terminates the NiFi cluster resources and billing activities while maintaining flow persistence.

  • Added support for resizing an existing CDF Deployment, which allows increasing and decreasing the memory and CPU allotted to an existing NiFi cluster

  • Added support for selecting the Node Storage Profile when creating a CDF Deployment. This allows the use of economical or high performance persistent storage for the NiFi cluster.

  • New flow deployments on AWS will now use GP3 volumes instead of GP2 providing a better cost/performance ratio.

  • Added active monitoring of NiFi cluster health for CDF Deployments which raises and resolves an alert automatically when a NiFi cluster encounters and recovers from issues impacting its running nodes.

  • Added ability to create sensitive dynamic properties in Flow Designer.

  • Diagnostic Bundles can now be requested via the Unified Diagnostics UI in the Cloudera Manager. Bundles will be uploaded directly to the selected support case.

  • Added Workspace Resources page to workload instances. This provides authorized users with a view of deployments, draft flows, inbound connections, and custom NAR configurations in the workspace.

  • Introduced new deployment alerts for Kubernetes component failures like NiFi pods or supporting statefulsets.

Changes and improvements

  • Support for updated Kubernetes server versions: AKS 1.26 / EKS 1.25.

  • Improved role assignment processing in the DataFlow workload application so role updates are automatically updated within 5 minutes by regular queries to FreeIPA. The user no longer has to log out and log back in to refresh their roles.

  • Overhauled NiFi node offloading behavior to be more robust and fault resistant in a variety of cases, including when offloading is disrupted by Kubernetes events.

  • Completed significant reliability improvements for larger scale CDF clusters up to 50 nodes, including a variety of efficiency refinements and support for vertical scaling for essential cluster resources.

  • Renamed existing Suspend Flow action to Stop Flow. Renamed the resulting deployment state Suspended to Flow Stopped. A CDF Deployment in Flow Stopped state does not remove NiFi cluster resources and the deployment remains billable.

  • Released the latest version of NiFi 1.23 which includes the following improvements and CDF integrations:

    • Kubernetes Leader Election and State Management - eliminates the need for a ZooKeeper pod in each CDF Deployment cluster.

  • Automatic restart the deployment under the hood if NiFi properties are changed.

  • Vertical auto scaling for Prometheus to automatically adjust memory to support a larger number of deployments and metrics.

  • Improved stability of helm upgrade during CDF upgrade process.

  • Retaining more NiFi provenance data given the current storage capacity allocated for that purpose.

Fixed issues

  • CDPDFX-7455: Fixed issue where underlying failure to create AKS cluster on CDF is masked by a 404 error. The actual Azure activity log failure is now raised in a status update event soon after the 404 failure occurs.

  • CDPDFX-7540: Fixed issue preventing Kafka to Kudu ReadyFlow from being opened in Flow Designer.

  • CDPDFX-7828: Improved sensitivity of CPU based autoscaling behavior when flow metrics autoscaling feature is enabled.

  • CDPDFX-7400: Fixed issue rendering an apostrophe in Flow Designer Data Viewer.

  • CDPDFX-7399: Fixed issue with MiNiFi pod failing to recover after a CDF upgrade encounters an issue and triggers rollback.

  • CDPDFX-7815: Fixed issue with validation of Custom NAR Configuration in RAZ enabled environments.

  • CDPDFX-7399: Fixed MiNiFi logging data migration.

  • CDPDFX-7704: Fixed racing condition among containers in DFX-Local pod.

  • CDPDFX-7737: Fixed issue where UDX collection is not associated with case properly.