Homepagecloud

Cloudera Data Flow on Cloud

Cloudera Data Flow is a cloud-native service that enables self-serve deployments of Apache NiFi data flows from a central catalog.

Cloudera Data Flow Deployments provides a cloud-native runtime to run your Apache NiFi flows through auto- scaling Kubernetes clusters, and centralized monitoring and alerting capabilities for the deployments.

Cloudera Data Flow Functions provides a cloud-native runtime to run your Apache NiFi flows as functions on the serverless compute services of AWS Lambda, Azure Functions, and Google Cloud Functions, targeting use cases that do not require always running NiFi flows.

Key features and benefits

Flow and resource isolation

Cloudera Data Flow allows you to easily isolate data flows from each other and guarantee a set of resources to each data flow without requiring administrators to create additional NiFi clusters. For each flow deployment, Cloudera Data Flow creates a dedicated, optionally auto-scaling NiFi cluster on the shared Kubernetes resources in a service. This way flow deployments can scale independently from each other, allowing you to isolate flow deployments and assign resources to deployments as needed.

Flow isolation can be useful when you want to guarantee a set of resources for a specific data flow or when you want to isolate failure domains.

Auto-scaling flow deployments

Cloudera Data Flow offers two types of auto-scaling capabilities for Apache NiFi data flows. Flow deployments may automatically scale up and down based on CPU utilization within the boundaries that are set in the deployment wizard. In this case Cloudera Data Flow scales flow deployments by adding or removing NiFi pods on the Kubernetes cluster as needed, as well as scaling the Kubernetes cluster up or down within boundaries specified during Cloudera Data Flow enablement.

On top of scaling based on CPU utilization, you may also enable Flow Metrics Scaling in the deployment wizard. This feature adds or removes NiFi pods to the Kubernetes cluster based on anticipated traffic on connection(s) where data first enters the flow. Scaling happens automatically, driven by a prediction algorithm, that uses a backpressure prediction metric to forecast how full a queue will be within a predefined period of time. The metric targets only connections which are attached to source processors. You do not need to configure anything during flow deployment. When Flow Metrics Scaling is enabled, both this metric and CPU utilization are considered. Whichever metric calls for higher scale will be obeyed.

Fault tolerant flow deployments

Flow deployments use persistent volumes to store NiFi repositories in a durable way. In case of an instance or pod failure, Cloudera Data Flow automatically spins up new pods and re-attaches the persistent volumes to ensure data processing continues from where it was interrupted.

Quick flow deployment with predefined ReadyFlows

You can quickly deploy a predefined set of data flows with minimal configuration called ReadyFlows. ReadyFlows provide you with an easy way to implement the most common data flow use cases.

Central monitoring dashboard and KPIs

You can monitor your flow deployments on a single dashboard. You can track important flow performance metrics by defining KPI alerts for your flow deployments.

Universal connectivity

You can connect to any data source or target using NiFi’s rich processor library, including on-premise data sources, cloud data storage, cloud data warehouses, log data sources, cloud data analytics services, or cloud business process services.

Flow designer

A no-code, self-service visual editor that allows you to build and test data flows.

Parameter groups

You can create groups of parameters and share them between data flows. Parameter groups allow you to centrally manage, share and reuse common parameters that your data flows depend on. When developing new data flows, developers and administrators alike can re-use these common parameters for a simplified development and deployment experience.

Deploying and managing Apache NiFi

  • Deploy NiFi easily and fast.
  • Manage multiple NiFi deployments.
  • Run multiple clusters with different NiFi 1 and 2 versions.
  • Run NiFi in a single-node mode or as a distributed cluster.

NiFi clusters without ZooKeeper

  • Enable NiFi deployment without ZooKeeper by handling leader election and state management directly within Kubernetes.
  • Store NiFi cluster state within Kubernetes configuration maps.

    Scaling NiFi clusters up and down

  • Manually scale up/down your NiFi deployments by adjusting the number of NiFi nodes in the configuration file.
  • Utilize Kubernetes Horizontal Pod Autoscaler (HPA) to monitor Central Processing Unit (CPU) utilization and trigger autoscaling based on resource consumption.

    Data durability

  • In-flight data remains uninterrupted even if a NiFi node goes down. Kubernetes will provision a new node and transfer the volumes from the failed node to the new one, ensuring continuous data processing.