Overlay networks for Azure environments in Cloudera Data Warehouse service
An overlay network is a software-defined layer of network abstraction that is used to run multiple separate, discrete virtualized network layers over the Azure virtual network. You use overlay networks to increase the number of IP addresses available to support nodes in your Cloudera Data Warehouse (CDW) cluster on Azure. This topic is about using overlay networks for CDW on Azure environments. It explains how they work in Azure and why to use them.
Default CDW networking
By default, CDW uses Azure Container Networking Interface (CNI). With Azure
CNI, every pod gets an IP address from the node subnet and is accessed directly. Each IP
address must be unique across your network space, and you must plan for them in advance of
deploying your CDW cluster. Each node has a configuration parameter for the maximum number of
pods that it can support. The equivalent number of IP addresses per node is reserved up front
for it. This requires advanced planning and it can often lead to IP address exhaustion. As an
alternative, you must rebuild the cluster in large subnets so your cluster can meet your
applications' demands. You can configure the maximum pods that are deployable to a node when
you create the cluster or when you create new node pools. However, if you do not specify the
maximum number of pods for the
maxPods property when you create new node
pools, by default each node gets 30 pods (with one IP address per pod.
About using the overlay network feature
To avoid IP address exhaustion, you can enable the Overlay Network feature when you activate an Azure environment to use with CDW. The Overlay Network feature uses kubenet networking for your Azure Kubernetes Service (AKS) clusters. For a full description of kubenet networking in AKS, see the Microsoft documentation.
How kubenet compares to Azure CNI
The following calculations compare kubenet to Azure CNI:
- kubenet: A simple /24 IP address range can support up to 251 nodes in the cluster. Keep in mind that Azure virtual network subnet reserves the first three IP addresses for management operations. 251 nodes can support up to 27,610 pods when you use a default maximum of 110 pods per node.
- Azure CNI: The same basic /24 IP address subnet range can only support a maximum of 8 nodes in the cluster. 8 nodes can only support up to 240 pods with a default maximum of 30 pods per node.
Guidelines for choosing the Overlay Network feature in CDW
The following guidelines help you to decide whether to enable the Overlay Network feature when you are activating Azure environments for CDW:
- Use the Overlay Network feature (kubenet) when the following conditions are true for
your CDW deployment:
- You have a limited IP address space.
- Most of the pod communication is within the cluster.
- You do not need advanced AKS features such as virtual nodes or to secure traffic between pods with network policies. See Microsoft documentation for more information on Azure network policies.
- Do not use the Overlay Network feature (Azure CNI) when the following is true for your
- You have adequate IP addresses available.
- Most of the pod communication is to resources outside of the cluster.
- You do not want to manage User Defined Routing (UDR) for connectivity between pods across nodes, which is required for kubenet.
- You do not need advanced AKS features as described above.
Limitations of the Overlay Network feature in CDW on Azure environments
If you choose the Overlay Network feature, which uses kubenet, when you activate Azure environments for CDW, the following limitations exist:
- Permissions must be assigned before cluster creation. Ensure that you are using a service principal with write permissions on your customer subnet and custom route table.
- Managed identities are not supported with custom route tables in kubenet.
- A custom route table must be associated to the subnet before you create the AKS cluster. Afterwards, this route table cannot be updated. All routing rules must be added or removed from the initial route table before you create the AKS cluster.
- All subnets within an AKS virtual network must use the same route table.
- Every AKS cluster must use a unique route table. You cannot re-use a route table with multiple clusters.
Node count limits
The following node count limits apply to whether you use the Overlay Network feature (kubenet) or not:
- When you use the Overlay Network feature that uses kubenet you can have up to 400 nodes/AKS cluster.
- Without it, which uses Azure CNI, you can have up to 1,000 nodes/AKS cluster.