Cloudera Docs

Cloudera Flow Management - Kubernetes Operator overview

Cloudera Flow Management (CFM) Operator allows you to deploy and manage NiFi clusters and NiFi Registry instances on Kubernetes application platforms. Cloudera Flow Management - Kubernetes Operator simplifies data collection, transformation, and delivery across your enterprise. Leveraging containerized infrastructure, the operator streamlines the orchestration of complex data flows.

Apache NiFi is the core data ingestion engine of the operator. NiFi is a robust tool for building data movement pipelines using a visual flow designer. With hundreds of built-in processors, it facilitates easy connections to various applications and allows for the transformation of data structures or formats as needed. Supporting both structured and unstructured data for streaming and batch integrations, NiFi is a fundamental component of modern data pipelines and is essential in multimodal Generative AI use cases.

Key capabilities

Deploying and managing NiFi

  • Deploy NiFi easily and fast.

  • Manage multiple NiFi deployments.

  • Run different NiFi versions within each cluster (NiFi 1.25, 2.0 [Technical Preview]).

  • Run NiFi in a single-node mode or as a distributed cluster.

Deploying and managing NiFi Registry

  • Deploy one or more NiFi Registries.

  • Connect a NiFi Registry instance to one or more NiFi clusters and use it for central flow versioning.

cfmctl command line tool for installation

  • Simplify the installation and deployment of Cloudera Flow Management - Kubernetes Operator and NiFi and NiFi Registry CRDs.

  • Quickly assess environment readiness.

NiFi clusters without ZooKeeper

  • Enable NiFi deployment without ZooKeeper by handling leader election and state management directly within Kubernetes.

  • Store NiFi cluster state within Kubernetes configuration maps.

Scaling NiFi clusters up and down

  • Manually scale up/down your NiFi deployments by adjusting the number of NiFi nodes in the configuration file.

  • Utilize Kubernetes Horizontal Pod Autoscaler (HPA) to monitor Central Processing Unit (CPU) utilization and trigger autoscaling based on resource consumption.

Data durability

  • In-flight data remains uninterrupted even if a NiFi node goes down. Kubernetes will provision a new node and transfer the volumes from the failed node to the new one, ensuring continuous data processing.

Full lifecycle management

  • Change your configuration by deploying a modified configuration file and let the operator reconcile.

  • Perform NiFi version upgrades effortlessly by specifying a new NiFi image. Support for rolling upgrades is coming soon for NiFi 2 clusters.

Deployment architecture

Cloudera Flow Management - Kubernetes Operator is responsible for managing NiFi and NiFi Registry deployments. It is deployed within a designated operator namespace, while the actual NiFi and NiFi Registry instances are managed within one or more separate namespaces. The following diagram shows a typical NiFi deployment with Cloudera Flow Management - Kubernetes Operator.

Licensing

Cloudera Flow Management - Kubernetes Operator requires a valid and active license to function. Licenses are made available to you together with your Cloudera credentials as part of your license and subscription agreement with Cloudera.

Licenses are registered during the installation of the Cloudera Flow Management - Kubernetes Operator and stored in a Kubernetes secret. They can be updated at any time.