Release NotesPDF version

What's new in Cloudera Flow Management 4.0.0 [Technical Preview]

Explore the new features and improvements in Cloudera Flow Management and learn how these additions can enhance your workflows.

Cloudera Flow Management 4.0.0 is available as a Technical Preview and is not supported for operational use.

For installation instructions, see Cloudera Flow Management installation workflow.

Cloudera Flow Management 4.0.0 is based on the General Availability (GA) version of Apache NiFi 2.0.0 This Technical Preview release introduces several new NiFi 2 features and Cloudera-exclusive enhancements, including a Python API for developing native NiFi components in Python. While NiFi 2 brings significant improvements, the transition from NiFi 1.x to 2.x also introduces breaking changes, making the upgrade process complex.

There is no supported upgrade path from Cloudera Flow Management versions using NiFi 1 (2.1.7 and lower) to Cloudera Flow Management 4.0.0.

To assist with future upgrades and flow migration, Cloudera will provide a Migration Tool, offering a semi-automated solution for migrating NiFi flows from version 1 to version 2. The tool will focus on adapting flows to NiFi 2 features, such as parameter contexts and new supported components. While it will not resolve all breaking changes, it will cover as many as possible to reduce the manual efforts needed and ensure better compatibility.

Here are the most important new features and improvements of this release:

Flow Analysis Tool

The Flow Analysis tool introduces a rules engine that provides real-time feedback during flow design, ensuring that flows adhere to best practices and configuration guidelines. This feature allows NiFi administrators to define a set of rules to enforce best practices in flow design in NiFi. The defined rules prevent invalid flows from being deployed to production, improving performance and compliance. It is fully integrated into the NiFi UI, offering an intuitive new view that can be customized based on customer requirements.

Flow Analysis Rules

Apache NiFi 2.0 introduces flow analysis rules, a new feature that enhances flow validation and management by evaluating components or parts of a flow. These rules can generate violations to help optimize or maintain flow design. For more information, see Supported NiFi flow analysis rules.

Python API and AI processors supporting Python

There is now a first-class citizen Python API allowing users to develop NiFi processors using Python. For more information, see the NiFi Python Developer’s Guide.

One of the key features introduced in Apache NiFi 2 is native support for Python processors. This capability allows you to create custom processors using Python, enabling seamless integration of Python scripts into your dataflows. With each milestone release of NiFi 2, Python integration continues to evolve, providing developers with enhanced functionality, greater flexibility, and more powerful tools for building robust dataflows.

The below list shows the Python processors that are available in Cloudera Flow Management 4.0.0.

  • Bedrock
  • ChunkData
  • ChunkDocument
  • EmbedData
  • InsertToMilvus
  • LexicalQueryMilvus
  • ParseDocument
  • PartitionCsv
  • PartitionDocx
  • PartitionHtml
  • PartitionPdf
  • PartitionText
  • PromptChatGPT
  • PutChroma
  • PutOpenSearchVector
  • PutPinecone
  • PutQdrant
  • QueryChroma
  • QueryOpenSearchVector
  • QueryPinecone
  • QueryQdrant
  • VectorQueryMilvus
Stateless Engine at Process Group level

It is now possible to configure a Process Group to use the Stateless Engine for running flows. This is particularly useful for transactional use cases such as Change Data Capture (CDC), or scenarios where a message broker is the source, aiming to achieve exactly-once semantics. For more information, see the Apache NiFi User Guide.

New NiFi components

Cloudera Flow Management 4.0.0 has introduced several new NiFi components. For a comprehensive list of supported NiFi components, see the Support Matrix.

Breaking changes

There are a number of breaking changes between NiFi 1 and NiFi 2. See Behavioral changes for more information about these changes. Additionally, you can anticipate further changes in the upcoming releases, particularly concerning components completely removed in favor of better and more efficient alternatives.