What's new in Cloudera Flow Management 4.0.0 [Technical Preview]

Explore the new features and improvements in Cloudera Flow Management and learn how these new additions can enhance your workflows.

Cloudera Flow Management 4.0.0 is a NiFi 2 Technical Preview based on the General Availability (GA) version of Apache NiFi 2.0.0. This release also includes several new NiFi 2 and Cloudera-exclusive features and improvements, including a powerful Python API for developing native NiFi components using Python. While this NiFi version brings significant enhancements, the transition from NiFi 1.x to NiFi 2.x comes with numerous breaking changes, potentially making the upgrade process complex and challenging.

Currently, there is no supported upgrade path from Cloudera Flow Management versions using NiFi 1 (2.1.7 and lower) to Cloudera Flow Management 4.0.0. Cloudera will provide tooling to help with upgrades and manage breaking changes in the future. The NiFi Migration Tool will offer a semi-automatic solution for migrating NiFi flows from 1.x to 2.x, focusing on compatibility with new NiFi 2 features, such as parameter contexts and new supported components. The tool will not address all breaking changes, but will aim to cover as many as possible during migration. It will reduce manual work and will ensure the best compatibility.

For installation instructions, see Cloudera Flow Management installation workflow.

Here are the most important new features and improvements of this release:

Flow Analysis Tool

The Flow Analysis tool introduces a rules engine that provides real-time feedback during flow design, ensuring that flows adhere to best practices and configuration guidelines. This feature allows NiFi administrators to define a set of rules to enforce best practices in flow design in NiFi. The defined rules prevent invalid flows from being deployed to production, improving performance and compliance. It is fully integrated into the NiFi UI, offering an intuitive new view that can be customized based on customer requirements.

Python API and AI processors supporting Python

There is now a first-class citizen Python API allowing users to develop NiFi processors using Python. For more information, see the NiFi Python Developer’s Guide.

One of the key features introduced in Apache NiFi 2 is native support for Python processors. This capability allows you to create custom processors using Python, enabling seamless integration of Python scripts into your dataflows. With each milestone release of NiFi 2, Python integration continues to evolve, providing developers with enhanced functionality, greater flexibility, and more powerful tools for building robust dataflows.

The below list shows the Python processors that are available in Cloudera Flow Management 4.0.0.

Bedrock
ChunkData
ChunkDocument
EmbedData
InsertToMilvus
LexicalQueryMilvus
ParseDocument
PartitionCsv
PartitionDocx
PartitionHtml
PartitionPdf
PartitionText
PromptChatGPT
PutChroma
PutOpenSearchVector
PutPinecone
PutQdrant
QueryChroma
QueryOpenSearchVector
QueryPinecone
QueryQdrant
VectorQueryMilvus

Stateless Engine at Process Group level

It is now possible to configure a Process Group to use the Stateless Engine for running flows. This is particularly useful for transactional use cases such as Change Data Capture (CDC), or scenarios where a message broker is the source, aiming to achieve exactly-once semantics. For more information, see the Apache NiFi User Guide.

New NiFi components

Cloudera Flow Management 4.0.0 has introduced several new NiFi components. For a comprehensive list of supported NiFi components, see the Support Matrix.

Breaking changes

There are a number of breaking changes between NiFi 1 and NiFi 2. See Behavioral changes for more information about these changes. Additionally, you can anticipate further changes in the upcoming releases, particularly concerning components completely removed in favor of better and more efficient alternatives.