Using Inbound Connections with an external load balancer
Once a dataflow deployment with an Inbound Connection Endpoint is available, you can go on and connect an external load balancer to start sending data.
Inbound Connection Endpoints are created in CDF with an internal Layer 4 (L4) load balancer (LB). Nevertheless, it is also possible to use your own native Layer 7 (L7) LB (Application Gateway on Azure, Application Load Balancer on AWS, respectively) in front of the Cloudera managed L4 LB.
Cloudera recommends achieving this by configuring your L7 LB to use the DataFlow
deployment LB as a backend. Enabling TLS between your LB and the DataFlow LB is recommended, but
mTLS is not possible for the backend connection. This means that your Listen Processor (e.g.,
ListenHTTP) in your NiFi flow cannot be configured with
Client Auth = Required
when using an external LB as a gateway. .
You may configure the listening side of your LB and routing rules according to the requirements of your organization.
Alternatively, you may be required to use a L4 LB provided by your organization in front of the Cloudera managed LB. This is also possible, although Cloudera recommends directly using the CDP managed L4 LB when possible.
Typically, when using an external load balancer to act as a gateway, the internal managed load balancer should stay private. This can be accomplished by deselecting the “Use Public Endpoint” option when enabling DataFlow for your environment, which limits DataFlow to only use private subnets for all resources. If public access is needed, that would be done by exposing private resources via the external gateway load balancer.
Currently, an Inbound Connection Endpoint can only be created during flow deployment, and cannot be reassigned without terminating the flow deployment for which it was created.
To configure an external load balancer, you need to go through the following steps: