VNet and subnets

When registering an Azure environment in CDP, you will be asked to select a VNet and one or more subnets.

You have two options:

  • Use your existing VNet and subnets for provisioning CDP resources.
  • Have CDP create a new VNet and subnets. All CDP resources will be provisioned into this new VNet and subnets.

Existing VNet and subnets

If you would like to use your own VNet, it needs to fulfill the following requirements:

  • The VNet has at least one subnet.
  • VNet should be able to make an outbound connection with the internet or set of CIDRs and ports provided by Cloudera.
  • If you would like to deploy Data Warehouse in your environment:
    • Make sure Azure VNet subnets are large enough to support the DW load. When an Azure environment is activated for DW service, an Azure Kubernetes Service (AKS) cluster is provisioned in your subscription. The AKS cluster uses the Azure Container Networking Interface (CNI) plug-in for Kubernetes. This plug-in assigns IP addresses for every pod running inside the Kubernetes cluster. By default, the maximum number of pods per node is 30. This means that you need approximately 3,200 IP addresses for a 99-node cluster. if you activate an environment for DW service, make sure that the subnets are large enough on the Azure VNet for the DW load. Cloudera recommends using a CIDR/20 subnet or larger.
    • Configure service endpoints on DW subnets. You must configure service endpoints on the subnets used for the DW service. This ensures that the network traffic between DW components and Azure services remain on the Microsoft Azure backbone network. To use with DW, Microsoft.Storage and Microsoft.SQL must be registered. Without this step DW service cannot be activated on existing Azure VNets. For more information, see Virtual Network service endpoints in the Azure documentation.
  • If you would like to deploy Machine Learning, note that each ML workspace requires its own subnet.

Verify the limits of the VNet and subnets available in your Azure subscription to ensure that you have enough resources to create clusters in CDP.

New VNet and subnets

If you would like CDP to create a new VNet, you will need to specify a valid CIDR in IPv4 range that will be used to define the range of private IPs for VM instances provisioned into these subnets. Default is 10.10.0.0/16. Consider changing the IP range to correspond to corporate policies for standardized IP address ranges. The CIDR must match the <network mask>/16 pattern.

By default CDP creates more than 30 subnets and divides the address space as follows:

  • 32 x /24 private subnet for ML
  • 3 x /19 private subnet for DW
  • 3 x /19 private subnet for Data Lake and Data Hub
  • 3 x /24 public subnet

You can disable creating private subnets, in which case only 3 public subnets will be created.

For more information about VNet and subnets, refer to VNet and subnet planning.