Cloudera Data Flow Architecture

Cloudera Data Flow follows a two-tier architecture where product capabilities like the Overview, Catalog and Environment management are hosted on the Cloudera Control Plane while the deployments running the flows that process your data are provisioned in a Cloudera environment which represents infrastructure in your cloud provider account. Learn more about the service architecture, and how Cloudera Data Flow enables the various service users to achieve their goals.

When you enable Cloudera Data Flow for one of your registered Cloudera environments, Cloudera Data Flow creates and configures the required infrastructure including a Kubernetes cluster, Kubernetes Operators and the Cloudera Data Flow workload application in your cloud account. After Cloudera Data Flow has been successfully enabled for an environment, users can deploy flow definitions into this environment. When deploying a flow definition, you can decide whether to create a dedicated NiFi cluster on Kubernetes allowing you to treat NiFi flows as isolated deployments, or to add the new flow to an existing deployment to optimize costs.

Deployments run the NiFi flow logic and process data in your cloud account. Therefore data that is being processed by a flow does not traverse the Cloudera Control Plane. Flows and deployments send heartbeats containing health and performance information to the Control Plane where this data is visualized and presented in the Deployments view.

The Cloudera Data Flow Functions feature allows you to deploy NiFi flows stored in the Cloudera Data Flow Catalog as functions executed within AWS Lambda, Azure Functions and/or Google Cloud Functions. Leveraging Cloudera Data Flow Functions does not require to have Cloudera Data Flow enabled in a Cloudera environment. When a Cloudera Data Flow function is executed, the function will interact with the Control Plane to retrieve the flow definition and to send monitoring information.

Learn more about the specific details of Cloudera Data Flow architecture from the diagram below.