Cloudera Docs

What's new in Cloudera Flow Management 4.10.0

Explore the new features and improvements in Cloudera Flow Management and learn how they can modernize your data pipeline development and operations.

Cloudera Flow Management 4.10.0 marks the General Availability (GA) release of Apache NiFi 2 within Cloudera’s data platform.

This release introduces a broad set of NiFi 2 changes along with Cloudera-exclusive capabilities and improvements, providing an enterprise-ready foundation for building, operating, and scaling your data pipelines. Switching to NiFi 2 in your data flow development and operation is not just a technical update, it is a modernization of the whole data flow experience, offering greater performance, flexibility, and security.

New features and changes

The following sections highlight the most important changes introduced with NiFi 2 and Cloudera Flow Management 4.10.0.

Native Python 3 support

NiFi 2 introduces a powerful Python API, enabling you to develop processors, controller services, and reporting tasks directly in Python. This allows you to embed Python-based orchestration and data manipulation into your flows without relying on external scripting.

AI & GenAI components

NiFi 2 provides a framework for building A-enabledI processors. Cloudera extends this with production-grade GenAI components, supporting advanced AI-driven workflows.

Flow Analysis Rules

A new, built-in rules engine enables real-time data quality validation within your flows. You can define rules to detect schema violations, missing fields, or implement custom business logic, all without additional coding.

Enhanced authentication

NiFi 2 removes legacy Kerberos configuration methods and enforces the use of Kerberos UserService. Authentication and encryption are now stricter and more secure by default.

Parameterization enhancement

Global variables have been replaced by a granular parameter framework, improving flow modularity and manageability.

Redesigned user interface

The new NiFi 2 UI is faster, cleaner, and more intuitive. It supports modular flow design with reusable components, speeding up development and allows for the centralization of design patterns in large projects.

Removed components

Over 140 components have been removed and must be replaced in NiFi 2. For more information on breaking changes between NiFi 1 and 2, see Behavioral changes.

Readded components

Cloudera Flow Management 4.10.0 restores support for several NiFi components that were deprecated in Apache NiFi 2. These components have been re-added to maintain compatibility and support key use cases. For more information, see Reintroduced NiFi components.

Migration tool

Cloudera provides a Flow Management Migration Tool that helps you replace deprecated processors, update configurations, and handle breaking changes automatically in your data flows. For more information, see Cloudera Flow Management Migration Tool.

Other improvements

NiFi 2 brings significant performance enhancements, especially important for high-volume workloads. It expands connectivity options both for source and target systems, simplifying to connect your data pipelines in hybrid environments. With richer native integration features, you can reduce reliance on custom processors. NiFi 2 also improves developer productivity with enhanced SDKs for Python and Java, enabling faster development and more efficient flow management.

New NiFi components

Cloudera Flow Management 4.10.0 introduces several new NiFi components to support broader integration and processing capabilities.

New processors:
  • GetBoxFileCollaborators
  • ExecuteSparkInteractive
  • GetBoxGroupMembers
  • ConsumeBoxEnterpriseEvents
  • SawmillTransformRecord
  • PutSolrRecord
  • CaptureChangeDebeziumMongoDB
  • SawmillTransformJSON
  • GetSolr
  • GetS3ObjectTags
  • FetchBoxFileRepresentation
  • PutSolrContentStream
  • ListBoxFileInfo
  • QuerySolr
  • ListenBeats
  • ConsumeBoxEvents
  • FetchBoxFileInfo
Python processors:
  • New
    • PromptClaude
    • TokenCount
    • PromptAzureOpenAI
    • PromptOpenAI
  • Renamed
    • Bedrock renamed to PromptBedrock
New controller services:
  • ClouderaEncodedSchemaReferenceReader
  • ClouderaEncodedSchemaReferenceWriter
  • PhoenixThickConnectionPool
  • PhoenixThinConnectionPool
  • ClouderaAttributeSchemaReferenceWriter
  • DeveloperBoxClientService
  • RESTCatalogService
  • ClouderaAttributeSchemaReferenceReader
  • LivySessionController
  • PEMEncodedSSLContextProvider
  • StandardDatabaseDialectService
New parameter provider:
  • PropertiesFileParameterProvider
Flow Analysis Rules:
  • New
    • RequireMergeBeforePutIceberg
    • RestrictFlowFileExpiration
  • Replaced
    • RestrictBackpressure replaced with RestrictBackpressureSettings

For the full list of supported NiFi components, see the Support Matrix.

Reintroduced NiFi components

Cloudera Flow Management 4.10.0 restores support for several NiFi components that were deprecated upstream in Apache NiFi 2.

Processors:
  • PutSolrContentStream
  • PutSolrRecord
  • QuerySolr
  • ExecuteSparkInteractive
Controller service:
  • LivySessionController

Breaking changes between NiFi 1 and NiFi 2

Cloudera Flow Management 4.10.0 incorporates key changes to maintain compatibility with Cloudera’s ecosystem while aligning with the open-source NiFi 2 project.So as part of the transition from NiFi 1 to NiFi 2, Cloudera has moved a large number of components downstream that were removed from the open-source Apache NiFi project. These adjustments aim to provide better support for Cloudera customers by aligning with specific Cloudera technologies and minimizing disruptive changes.

For example:

  • Hive components: NiFi in Cloudera Flow Management includes Hive components built against Cloudera’s version of Hive, which has diverged from Apache Hive, ensuring compatibility with Cloudera’s ecosystem.

  • Kafka components: Cloudera has retained Kafka 2.6 components in Cloudera Flow Management to facilitate smoother transitions from legacy Kafka components to the new Kafka architecture based on controller services, reducing the risk of breaking changes.

  • Jython scripting: Jython scripting support remains available for certain scripting components, despite being removed upstream.

For more information on these breaking changes, see Behavioral changes.

Upgrade and migration options

There is no supported in-place upgrade path from Cloudera Flow Management versions powered by NiFi 1 (2.1.7 and earlier) to Cloudera Flow Management 4.10.0.

Cloudera provides a Migration Tool that automates complex, repetitive, and error-prone manual tasks in updating flow configurations, reducing manual effort and ensuring compatibility with NiFi 2 features.

This Cloudera Flow Management Migration Tool simplifies the transition by:
  • Replacing removed processors
  • Converting variable-based configurations to parameters
  • Reconfiguring flows to use new controller services
  • Converting older templates into flow definitions
  • Adapting to security, data type, and API changes introduced in NiFi 2

For more information, see the Cloudera Flow Management Migration Tool documentation.

If you previously installed Cloudera Flow Management 4.0.0 [Technical Preview], you can upgrade to 4.10.0 directly. For instructions, see Upgrading from Cloudera Flow Management 4.0.0 to 4.10.0,

Installation

To install Cloudera Flow Management 4.10.0 from scratch, follow the Cloudera Flow Management installation workflow.