Azure Load Balancers in Data Lakes and Data Hubs
The Azure Load Balancer is used in multiple places in CDP Data Lakes and Data Hubs. It is used as a frontend for Knox in both Data Lakes and Data Hubs, and for Oozie HA in HA Data Hubs.
|CDP component||Azure Load Balancer use case|
|Data Lake||A load balancer is configured in front of Knox for Data Lakes of all shapes.|
|HA Data Hub||A load balancer is configured for all Data Hubs created from a default template where
Knox and/or Oozie is running in HA mode.
This can be overridden by setting the “enableLoadBalancer” setting in a custom template to “false”.
|An environment with Public Endpoint Access Gateway enabled||When the Endpoint Gateway is enabled, Load balancers are created in front of Knox for all Data Lakes and Data Hubs attached to the environment.|
In the event that a Data Lake or Data Hub uses private networks (meaning the “Create Public IPs” option is disabled during environment creation and the Public Endpoint Access Gateway is not enabled), an internal load balancer is created for ingress traffic to Knox in all Data Lakes and in Knox HA Data Hubs running in that environment.
Because CDP uses a Standard SKU Azure Load Balancer, the internal load balancer does not allow public egress. To allow public egress, a secondary public load balancer is created. The public egress load balancer has only outbound rules defined, and does not handle ingress traffic.