VPC and subnets

When registering an AWS environment in Cloudera, you will be asked to select a VPC and two or more subnets. You can use your existing VPC and subnets for provisioning Cloudera resources.

You need to verify the limits of the VPC and subnets available in your AWS account to ensure that you have enough resources to create clusters in Cloudera.

If you would like to use your own AWS VPC, it must meet the following requirements:

The VPC has at least two subnets, each in a different availability zone.

If you plan to expose your cluster via a public load balancer, you need to have at least one public and one private subnet in the same availability zone and your cluster must be deployed to one of those subnets.

The VPC subnets must be connected to an Internet Gateway OR a NAT Gateway. The VPC should be able to make an outbound connection with the internet or set of CIDRs and ports provided by Cloudera.

Cloudera supports public subnets and private subnets. For private subnets, you must enable Cluster Connectivity Manager.

Ensure the CIDR block for the subnets is sized appropriately. In general there is no way to increase the subnet size without recreating the environment and VPC, although Cloudera Data Warehouse service allows you to use overlay networks.

When using Cluster Connectivity Manager on AWS, you have an option to use Public Endpoint Access Gateway. If you would like to use it, you should have at least 2 public subnets in the VPC. The exact number of public subnets should be equal to the number of private subnets. Furthermore, availability zones of the public and private subnets must match.

Depending on the Cloudera services that you are planning to use, you may also need the following:

If you are planning to use Cloudera DataFlow, Cloudera Data Engineering, Cloudera Data Warehouse, or Cloudera AI you must enable Amazon DNS with the VPC. Corporate DNS is not supported. For guidelines on how verify your DNS settings, refer to sections 1-3 in AWS environment requirements checklist for the Cloudera Data Warehouse service.

If you are planning to use Cloudera DataFlow, Cloudera Data Engineering or Cloudera AI, you must tag the VPC and the subnets as shared so that Kubernetes can find them. For load balancers to be able to choose the subnets correctly, you are also required to tag private subnets with the kubernetes.io/role/internal-elb:1 tag, and public subnets with the kubernetes.io/role/elb:1 tag.

If you want to use the private network feature with Cloudera Data Warehouse service, you must have three public subnets and three private subnets in your VPC.

If you are planning to use the Cloudera Data Warehouse service, you must verify that the VPC has the settings listed in the AWS environment requirements checklist for the Cloudera Data Warehouse service.

If you plan to use the private networking feature in the Cloudera Data Warehouse service, refer to Supported deployment modes for private networking in AWS and Prerequisites for private networking in AWS environments.

If you are planning to use Cloudera DataFlow, Cloudera Data Engineering, Cloudera Data Warehouse, or Cloudera AI, you may also want to review the following AWS documentation: Amazon EKS - Cluster VPC Considerations and Creating a VPC for your Amazon EKS Cluster.

VPC and subnets

We want your opinion

How can we improve this page?