DataFlow Networking

DataFlow supports different networking options depending on how you have set up your VPC and subnets. If you want DataFlow to use specific subnets, make sure that you specify them when registering a CDP environment.

If you specified a mix of public and private subnets during environment registration, DataFlow by default will provision the Kubernetes nodes in the private subnets. For DataFlow to work, the private subnets require outbound internet access. This can be achieved by configuring NAT gateways in separate public subnets and making sure that outbound internet traffic is routed via the NAT gateway. The VPC you’re using needs to have an Internet Gateway set up which ultimately provides internet access to the public subnets. Following this approach allows the DataFlow services running on Kubernetes nodes in your private network to connect to the internet while also preventing inbound connections from the internet.

You can configure DataFlow to either use a private or public load balancer to allow users to connect to flow deployments. Using a private load balancer is possible when your CDP environment contains at least two private subnets. When you’re using a private load balancer, you need to ensure connectivity between the client network from where your users are initiating connections and the private subnets in your VPC. This is typically done by setting up VPN access between the private subnets in AWS and the corporate network.

If you want to allow users to connect to flow deployments from the internet you can use the public load balancer option. This option will provision public load balancers in public subnets allowing your users to connect to flow deployments without the need to set up VPN connectivity between the private subnets and your corporate network.

Cloudera recommends that you either use a fully private deployment in private subnets with private load balancers or a mix of private subnets with a public load balancer. Cloudera does not recommend provisioning DataFlow in public subnets.

The image below represents a fully private deployment where Kubernetes nodes and load balancers are deployed in the private subnets.