What's new in Cloudera Flow Management 4.13.0

Explore the latest features and improvements in Cloudera Flow Management and how they support modern data pipeline development and operations.

Cloudera Flow Management 4.13.0 is based on Apache NiFi 2.11.0 and runs with Cloudera Manager on Cloudera on premises 7.3.2 SP1.

This release expands support for Apache Iceberg version 3 tables, adds integrations for cloud services and external systems, improves metrics reporting, and extends platform support. It also includes fixes for multiple Common Vulnerabilities and Exposures (CVEs).

The following sections highlight the main changes.

Rebase to Apache NiFi 2.11

This release updates the underlying Apache NiFi version to 2.11, bringing upstream fixes and improvements to the NiFi core and its components. For the complete list of components supported in Cloudera Flow Management 4.13.0, see the Support Matrix.

Platform support

Red Hat Enterprise Linux 9.6: Cloudera Flow Management 4.13.0 adds support for RHEL 9.6.
Spring Framework 7: The cdf-extensions dependencies have been updated to Spring Framework 7.

Apache Iceberg version 3 support

Cloudera Flow Management 4.13.0 upgrades Apache Iceberg to version 1.10.1 and adds support for the following Iceberg version 3 capabilities:

Deletion vectors: Process position deletes stored as deletion vectors in Iceberg tables.
Variant data type: Ingest and process semi-structured data represented using the Iceberg variant type.
Nanosecond timestamp precision: Preserve nanosecond precision when processing supported timestamp values.
Column default values: Apply Iceberg schema defaults when records omit fields that have defined default values.

The PutIceberg and PutIcebergCDC processors can also write to a specified Iceberg branch. This supports workflows that ingest data into a staging or validation branch before it is merged into the main branch.

New NiFi components

Processors

PutSnowflakeStreaming

It uses the Snowflake Ingest SDK and Snowpipe Streaming API to stream data directly from NiFi dataflows into Snowflake tables without intermediate staging files. It supports high-throughput, low-latency ingestion for pipelines that require lower latency than batch ingestion.

For limitations, see Authentication, SSL, and proxy limitations in PutSnowflakeStreaming.

EnrichGraphRecord

It enables you to build graph-based dataflows by creating or updating graph nodes and relationships and enriching their properties with incoming record data.

The updated GraphClientService API generates database-specific statements, such as Cypher, without requiring users to write queries manually.

Graph component types and labels allow the processor to take advantage of graph database indexes.

Parameter provider

ClouderaOnCloudParameterProvider

You can discover supported service connection details in Cloudera on cloud and populate NiFi Parameter Contexts with the help of this parameter provider reducing manual configuration needs. With the required environment credentials and endpoint information, the provider queries the cloud APIs to discover supported resources, including Data Hub clusters, Cloudera Data Warehouse virtual warehouse JDBC endpoints, Cloudera Operational Database connection details, and Kafka broker endpoints.

The provider automatically organizes the discovered details into parameter groups to simplify the configuration of NiFi processors and Controller Services.

Reporting task

OpenTelemetryReportingTask

It exports NiFi operational, flow, and JVM metrics to OpenTelemetry collectors and compatible observability platforms using the OpenTelemetry Protocol (OTLP).

It supports unauthenticated endpoints, Basic and Bearer token authentication, and secure connections through TLS or mutual TLS (mTLS).

Flow analysis rule

RequireLoadBalancedConnectionAfterPrimaryNodeSourceProcessor: This new flow analysis rule flags connections without load balancing downstream of source processors configured to run only on the Primary Node. It helps identify flow designs that could concentrate processing on a single node instead of distributing FlowFiles across the cluster.

For a full list of supported NiFi components, see the Support Matrix.

Enhancements to existing components and services

JSON Schema support in Cloudera Schema Registry

Cloudera Flow Management 4.13.0 adds JSON Schema support to the ClouderaSchemaRegistry Controller Service. You can register and retrieve JSON Schemas to validate records. Apache Avro schema support remains available.

The service also supports OAuth authentication through an OAuth2AccessTokenProvider Controller Service, providing an alternative to Kerberos or Basic authentication. OAuth support is also available for Confluent Schema Registry.

Apache Atlas lineage for Iceberg

The Atlas Reporting Task now analyzes provenance events from Iceberg processors, allowing their data lineage to be tracked in Apache Atlas.

Kerberos authentication for SMB

SmbjClientProviderService now supports KerberosUserService, enabling Kerberos authentication when accessing SMB shares.

External password sources for Impala

The ImpalaConnectionPool controller service has a new Password Source property for resolving passwords dynamically, including from supported external secret providers.

TLS for NiFi health metrics

Communication between NiFi and Cloudera Managerfor service health and status metrics now supports TLS encryption.

JSO

Flow Migration Tool

The available Cloudera Flow Migration Tool versions do not support migration to Cloudera Flow Management 4.13.0. A new release of the Migration Tool is in development to support this migration target.

For currently supported migration paths, see the Migration Tool documentation.

Upgrade and migration options

Installation

To perform a fresh installation of Cloudera Flow Management 4.13.0, follow the Cloudera Flow Management installation workflow.