VNet and subnets

When registering an Azure environment in Cloudera, you will be asked to select a VNet and one or more subnets.

You have two options:

Use your existing VNet and subnets for provisioning Cloudera resources.
Have Cloudera create a new VNet and subnets. All Cloudera resources will be provisioned into this new VNet and subnets.

Existing VNet and subnets

If you would like to use your own VNet, it needs to fulfill the following requirements:

The VNet has at least one subnet. If you are planning to run more than Cloudera Data Hub, you need additional subnets as specified in VNet and subnet planning.

VNet should be able to make an outbound connection with the internet or set of CIDRs and ports provided by Cloudera.

If you would like to use private Flexible Server with delegated subnet as your Azure database server type, then you should delegate a subnet to it, as described in Private setup for Azure Flexible Server. When using Private Link, a delegated subnet is not required.

If you would like to deploy Cloudera Data Warehouse in your environment, make sure Azure VNet subnets are large enough to support the Cloudera Data Warehouse load. When an Azure environment is activated for Cloudera Data Warehouse service, an Azure Kubernetes Service (AKS) cluster is provisioned in your subscription. The AKS cluster uses the Azure Container Networking Interface (CNI) plug-in for Kubernetes. This plug-in assigns IP addresses for every pod running inside the Kubernetes cluster. By default, the maximum number of pods per node is 30. This means that you need approximately 3,200 IP addresses for a 99-node cluster. if you activate an environment for Cloudera Data Warehouse service, make sure that the subnets are large enough on the Azure VNet for the Cloudera Data Warehouse load. Cloudera recommends using a CIDR/20 subnet or larger.

Configure service endpoints as described in Set up service endpoint for network. If you would like to use private endpoints instead of service endpoints for Data Lake and Cloudera Data Hub, meet the requirements described in Private endpoint for PostgreSQL; However, if you would like to use the Cloudera Data Warehouse service, you still need service endpoints regardless of what you choose for Data Lake and Cloudera Data Hub.

If you would like to deploy Cloudera AI, note that each Cloudera AI workbench requires its own subnet.

If you would like to deploy Cloudera Data Engineering, note that each Cloudera Data Engineering Service requires its own subnet.

If you would like to deploy Cloudera DataFlow, note that the Cloudera DataFlow service requires its own subnet. You can have only one Cloudera DataFlow service per environment.

Verify the limits of the VNet and subnets available in your Azure subscription to ensure that you have enough resources to create clusters in CDP.

VNets can be created and managed from the Azure Portal > Virtual Networks. For detailed instructions on how to create a new VNet on Azure, refer to Create a virtual network using the Azure portal in Azure documentation.

Egress connectivity for existing VNets and subnets

When you deploy an environment with an existing network of your own configuration, it is your responsibility to create egress connectivity for the required subnets in your VNet. Egress connectivity can be accomplished through a NAT gateway setup or user-defined routing. Alternatively you can create a secondary load balancer for public egress. See Azure Load Balancers in Data Lake and Data Hub for more information.

New VNet and subnets

If you would like Cloudera to create a new VNet, you will need to specify a valid CIDR in IPv4 range that will be used to define the range of private IPs for VM instances provisioned into these subnets. Default is 10.10.0.0/16. Consider changing the IP range to correspond to corporate policies for standardized IP address ranges. The CIDR must match the <network mask>/16 pattern.

By default Cloudera creates more than 30 subnets and divides the address space as follows:

3 x /24 public subnets for Data Lake and Cloudera Data Hub
3 x /19 private subnets for Cloudera Data Warehouse
32 x /24 private subnets for Cloudera AI, Cloudera Data Engineering, and Cloudera DataFlow
3 x /19 private subnets reserved for future use

You can disable creating private subnets, in which case only 3 public subnets will be created.

If you would like to use private Flexible Server with delegated subnet as your Azure database server type in private service mode, then you should delegate a subnet to it, as described in Private setup for Azure Flexible Server. Cloudera does not create the delegated subnet for you. When using Private Link, a delegated subnet is not required.

For more information about VNet and subnets, refer to the following VNet and subnet planning documentation.

Egress connectivity for new VNets and subnets

If you are creating a new network during environment registration, Cloudera ensures that egress connectivity is available. If the "Create Public IPs" option and Public Endpoint Access Gateway are disabled in your network, a separate load balancer is created for egress, though this load balancer requires certain public IP permissions that are granted as part of the required Azure permissions. If either "Create Public IPs" or Public Endpoint Access Gateway is enabled, then a public load balancer is created to handle both public ingress to port 443 and public egress.

Azure Load Balancers in Data Lake and Cloudera Data Hub for more information.